LLM-42: How Microsoft Made LLM Inference Reproducible Without Killing Throughput

Run the same prompt through a production LLM server twice. If the server uses dynamic batching, you may get different output on each run, even with temperature set to zero. Not because of intentional sampling randomness, but because of how GPUs compute sums. Microsoft Research’s LLM-42, accepted to SOSP 2026, solves this problem with a mechanism called verified speculation: a decode-verify-rollback loop that enforces determinism without abandoning dynamic batching or rewriting GPU kernels. The paper is by Raja Gond and Aditya K Kamath (University of Washington), Ramachandran Ramjee, and Ashish Panwar, all affiliated with Microsoft Research India, with the preprint available at arXiv:2601.17768.

Why the Same Prompt Produces Different Outputs

The non-determinism LLM-42 targets is not random sampling. It is a property of IEEE 754 floating-point arithmetic: addition is not associative. Adding the same set of numbers in different orders produces results that differ at the least-significant bits. GPU reduction kernels aggregate values across thousands of threads in parallel. The exact reduction order varies with the batch size, because different batch shapes trigger different kernel configurations.

Dynamic batching, a standard technique for efficient LLM serving, groups requests together as they arrive. Each new batch may have a different composition, which changes the kernel’s internal reduction ordering, which changes the floating-point rounding behavior. Over enough tokens, these tiny differences accumulate and produce a different token at some point in the sequence. From that point, the sequence diverges completely because LLM decoding is autoregressive: each token is conditioned on all previous tokens.

This is distinct from sampling randomness and invisible to most teams until they try to build systems that depend on reproducible outputs. The problem surfaces in compliance-regulated systems that need audit trails, in caching layers that assume identical inputs produce identical outputs, in debugging workflows that require bisecting a model’s behavior across runs, and in any A/B test that compares LLM output directly.

Two prior approaches exist. The first is to disable dynamic batching entirely and serve every request in its own fixed batch. This eliminates the batch-shape variation that drives the non-determinism, but the authors note it “severely degrades throughput.” The second approach is to make GPU kernels batch-invariant, meaning they produce identical results regardless of batch shape. This works but “tightly couples determinism to kernel design, requiring new implementations” and “imposes fixed runtime overheads, regardless of how much of the workload actually requires determinism.” Teams that want only some requests to be deterministic still pay the full cost for all of them.

The Decode-Verify-Rollback Protocol

LLM-42 takes a different path, one borrowed structurally from speculative decoding. The core observation is that non-deterministic divergence is rare in practice. Most GPU kernels already happen to use shape-consistent reductions, meaning that even with dynamic batching, most tokens produced on the fast path will match what a fixed-shape kernel would have produced. The second observation is that once a sequence has diverged once, the next token is still likely to be deterministic, since the divergence event does not permanently corrupt the decoding state.

Given these two observations, LLM-42 structures inference as three stages that form a loop:

Decode: The server generates tokens using the existing, unmodified SGLang kernels with full dynamic batching enabled. A window of 64 tokens (configurable via --llm42-window-size) is produced at full throughput before any verification begins.

Verify: The window of candidate tokens is replayed under a fixed-shape reduction schedule. Each candidate token is checked against what the fixed-shape kernel would have produced. The verifier groups this work into batches of 8 requests at a time (configurable via --llm42-verify-batch-size) to amortize the overhead across multiple concurrent deterministic requests.

Rollback: Any token that does not match the fixed-shape result is discarded. The system rolls back to the last verified consistent position and resumes decoding from there. Tokens that pass verification are committed and delivered to the client.

The key engineering decision here is what gets committed. The verifier only commits tokens that are “guaranteed to be consistent across runs,” meaning a second run of the same request under the same conditions will produce the same committed token. The rollback discards any tokens that cannot meet this guarantee. This is a stronger property than “usually the same” or “same on average.”

Per-Request Opt-In and Mixed Workloads

Not every application needs deterministic outputs. A chatbot probably does not care. A compliance system auditing financial decisions does. LLM-42 handles this with a per-request flag: requests marked is_deterministic=True enter the DVR loop, and all others continue through the standard non-deterministic fast path at full speed.

This is the design choice that separates LLM-42 from batch-invariant kernel approaches. Batch-invariant kernels impose their overhead on every request regardless of whether that request needs determinism. LLM-42 described it this way: it “incurs overhead only in proportion to the traffic that requires determinism.” A deployment where 10% of requests need auditing and 90% do not pays only 10% of the determinism overhead, not 100%.

The configuration the authors tested uses four NVIDIA H100 PCIe GPUs (80 GB HBM3 each), a 64-core CPU, and approximately 1.65 TB of DRAM. The implementation is built on SGLang v0.5.3, and the repository includes the core DVR logic in python/sglang/srt/llm42/, batch-invariant kernel wrappers in batch_invariant_ops/, and a complete benchmark suite in llm42_benchmarks/.

What the SOSP Artifact Evaluation Confirms

Conference artifact evaluations are an important but underappreciated signal. The SOSP 2026 artifact committee awarded LLM-42 all three badges: Available (code is publicly accessible), Functional (it runs and produces results), and Reproduced (independent reviewers ran the experiments and confirmed the paper’s claims). That third badge is rare. It means the determinism guarantee is not just a theoretical property described in the paper, but something external evaluators actually verified on hardware they controlled.

The related work from Perplexity searches surfaced a parallel line of research worth noting: Lossless but Not Free: An Empirical Anatomy of Speculative Decoding (arXiv:2607.17283) empirically validated that greedy speculative decoding produces bit-identical outputs compared to standard greedy decoding across K values of 1, 2, 4, and 8 speculation tokens. Their two-sample statistical test returned p=0.976, and real-model comparisons covered roughly 9,200 tokens. This matters for LLM-42 because verified speculation is the structural model both papers use: generate a candidate token sequence quickly, then check it against a reference. LLM-42 applies this to enforce cross-run consistency rather than to speed up inference.

A second parallel line of work, Bit-Exact AI Inference Verification Without Performance Tradeoffs (arXiv:2606.00279), takes a different mechanism entirely: software-only emulation of LLM inference designed to produce bit-identical results across different NVIDIA GPU variants. Where LLM-42 operates at the scheduling level (controlling reduction order through fixed-shape verification), the bit-exact emulation approach works at the arithmetic level (replacing hardware floating-point with emulated bit-reproducible arithmetic). The two are complementary, not competing: LLM-42 solves cross-run determinism within one hardware environment, while bit-exact emulation targets cross-hardware reproducibility.

Limitations and Open Questions

LLM-42 enforces what the paper calls token-level consistency: the same token sequence is produced across runs. This is weaker than bit-exact equality at the logit level. Two runs will produce the same discrete token, but the underlying probability distributions may differ by floating-point rounding in ways the rollback mechanism does not detect.

The rollback overhead is proportional to the mismatch rate. When a token fails verification, the system must discard the entire candidate window and regenerate from the rollback point. If the mismatch rate is high, this could approach the cost of running the full fixed-batch baseline. The paper’s claim that overhead is proportional to deterministic traffic assumes mismatch rates stay low, which the SOSP artifact evaluation confirmed for their tested configurations but which has not been tested across all models or all hardware configurations.

The current implementation builds on SGLang v0.5.3 and targets NVIDIA hardware. Portability to AMD, Intel, or other GPU vendors would require either new batch-invariant kernel wrappers or confirmation that those platforms’ reduction orders match the verification schedule. The window size and verify batch size are tunable parameters whose optimal values will vary by model, hardware, and traffic mix.

What This Means for Engineering Teams

The practical consequences are concrete. Teams building on LLM inference infrastructure can now treat determinism as a serving property rather than a model property. Instead of running a separate low-throughput deterministic server alongside a high-throughput production server, a single LLM-42 deployment handles both populations of requests in the same cluster. The throughput cost is paid only by the requests that opt in.

For compliance use cases, this changes the audit architecture: the serving layer can guarantee identical outputs for identical inputs under identical conditions, making the LLM comparable to a database query in terms of reproducibility. For LLM-based products that use output caching, determinism is now a dependency that the serving layer can actually satisfy rather than a property teams have to approximate through other mechanisms. For debugging, the ability to reproduce a specific output without spinning up a fixed-batch server simplifies root-cause analysis substantially.

The artifact evaluation infrastructure in the repository includes llm42_benchmarks/ for throughput measurements, test_batch_invariance/ for determinism verification, and Docker-based setup scripts. Teams evaluating LLM-42 for production have a working baseline to start from.

Key Takeaways

• LLM-42 from Microsoft Research India (SOSP 2026) introduces a decode-verify-rollback protocol that enforces token-level determinism without disabling dynamic batching.
• The root cause of non-determinism is floating-point non-associativity in GPU reduction kernels, triggered by batch-shape changes in dynamic batching.
• Only requests marked is_deterministic=True pay the verification overhead; mixed workloads run at full throughput for non-deterministic requests.
• The default window size is 64 tokens decoded between verification passes, with groups of 8 requests verified together per pass.
• SOSP 2026 artifact committee awarded Available, Functional, and Reproduced badges, meaning independent reviewers confirmed the determinism guarantee on external hardware.
• Parallel research (arXiv:2606.00279) addresses the harder cross-hardware bit-exact problem; LLM-42 focuses on cross-run consistency within one hardware configuration.

Work With Origins AI

Origins AI builds production AI systems for engineering teams. If deterministic inference is a requirement for your compliance, caching, or audit architecture, talk to our team.