Deployment Is the Unit of Reproducibility for Greedy LLM Inference
Even with constant weights, speed, and GPU, fresh vLLM deployments matched only 59–64% of the outputs on the GSM8K benchmark. The batch caps of 1, 8, and 32 behaved identically. Two fresh containers that shared the same compiled autotune state cache produced 1,319 identical outputs on different H100s. FP8 W8A8 had zero divergences in 13,190 comparisons. The text churn was close to 40%. The verdict flips were close to 1%. Accuracy was ≤ 0.23%. Most inference-configuration differences yielded the same 55 fragile questions, so reproducibility was determined by the deployment stack rather than the sampler.
The contradiction
Temperature-zero (greedy) decoding with a fixed seed is generally assumed to produce reproducible evaluations. In this experiment, three passes over the 1,319 GSM8K questions within a single vLLM engine yielded token-identical outputs. Yet, using identical configurations in separate deployments resulted in divergent outputs in 36–41% of cases.
The aggregate view completely hides the underlying variability. Across our experimental grid, accuracy stayed within a narrow 0.35-point range (94.87% to 95.22%). Two fresh deployments with the same configuration disagreed on around 500 outputs and flipped 11 to 17 verdicts.
Deployment in this paper refers to a single inference engine process running in a fresh container. It includes the inference engine version, the model weights, and the compiled and auto-tuned kernel state that is set up during initialization. Bit-wise identical means that the output token identities are equal.
Experimental setup
I held the evaluation conditions frozen across experiments. I used the GSM8K test split, a prompt template ending with "the final answer is N" and the same extractor for all backends. Answer line regex, followed by last number fallback. Greedy decoding, temperature: 0, Seed: 0, Max new tokens: 2,048, Thinking off (verified per run) and prefix caching off. I evaluated three different variations of Qwen/Qwen3.5-9B, a hybrid gated delta net and full attention model: the official BF16 checkpoint, an FP8 version quantized by vLLM through dynamic W8A8, and Intel's pinned INT4 AutoRound checkpoint. There were two isolated axes: the weight format (at cap 32) and BF16 batch cap with max_num_seqs of 1/8/32. We ran 23 experiments, each in a new container with fresh compile/autotune state while sharing deterministic FlashInfer JIT artifacts, spread across 22 separate H100s with the same SKU and driver. I also performed an in-process determinism probe, a shared-cache ablation, an HF transformers batch size 1 anchor, and an lm-evaluation-harness cross-check (94% vs our 95.2%).
Determinism ends at the deployment boundary
Generations were repeatable within a single deployment. Three GSM8K passes through one vLLM engine produced 1,319/1,319 bitwise identical pairs. Across fresh deployments, this was not observed:
| cell | reps | accuracy mean±sd | unstable questions | pairwise verdict flips | pairwise output identity |
|---|---|---|---|---|---|
| BF16, cap 1 | 3 | 94.87% ± 0.23 | 16 | 0.81% | 59.3% |
| BF16, cap 8 | 5 | 94.95% ± 0.20 | 25 | 0.91% | 59.3% |
| BF16, cap 32 | 5 | 95.09% ± 0.17 | 22 | 0.83% | 64.2% |
| FP8, cap 32 | 5 | 95.22% ± 0.00 | 0 | 0.00% | 100.0% |
| INT4, cap 32 | 5 | 95.21% ± 0.11 | 34 | 1.26% | 59.9% |
Table 1. Full GSM8K, one fresh container per repeat. "Unstable questions": verdict not constant across the cell's repeats (grows with repeat count). Flips and identity are means over a cell's run pairs.
I initially hypothesized that the batch cap would drive the observed churn since continuous batching changes which requests shared a kernel step. Results don't support this: cap 1, with no cross-request batching at all, churns like cap 32 (flip rate 0.81% vs 0.83%). The source of churn seems to predate batching.
FP8 is the outlier in the opposite direction. Across 13,190 paired comparisons spanning five H100s, we observed zero divergent outputs. This is not an inherent property of FP8, but just a no-divergence observation under our conditions (Qwen3.5-9B, vLLM 0.25.0's online W8A8, H100).
Figure 1. Accuracy per run and cell mean, above per-pair verdict-flip rates; the HF line is a single reference run.
One shared cache made two deployments bitwise-identical
If deployment-specific compiled kernel state is the source of divergence, then transplanting the state should reproduce the outputs. Rep 0 built the torch.compile/autotune cache, cold and sterilized it; reps 1 and 2, fresh containers on 2 H100s, loaded the shared cache. The loaders produced 1,319/1,319 identical outputs with zero flips and identical accuracy. By contrast, the builder versus either of the loaders demonstrated standard fresh deployment signatures: 793/1,319 identical outputs with 12 verdict flips. Building a cache and loading an existing one result in distinct deployment draws.
Under the BF16 config, loading the same compiled artifacts removed cross-deployment divergence, even across separate H100s. What varies inside the cache is being inferred but not isolated. The engine configuration enables timing-based auto-tuning (benchmark_combo_kernel: True), which may select different kernel variants under timing noise. We did not compare the different cache contents, and we are interpreting one loader pair from one build.
Figure 2. Pairwise output identity: ten reference pairs (median 60.4%), builder–loader 793/1,319, loader–loader 1,319/1,319 across physical cards.
The churn compresses as it propagates
Divergence is both front-loaded and widespread, with first differences around token 103 to 110 and broad, 133/1,319 questions diverged in same config pair. The resulting impact on benchmark outcomes is limited: in BF16 cap 32, 4,724/13,190 pair-questions diverged but produced only 110 verdict flips, a 43× attenuation (32–50× across cells). 98–99% of divergent pairs still reach the same final answer, and the surviving flips partly cancel into a ≤0.23-point accuracy standard deviation.
Every within-configuration verdict flip occurred in 55 out of 1,319 questions (4.2%). Loaded with problems that oscillate among a few internally consistent questions. For example, Q607 supports three readings of a pump capacity phrase, and the arms wander among them:
- BF16 alternates between 10 and 20.
- INT4 between 10 and 40.
- FP8 consistently produces the reference answer of 10.
Hence, the deployment-induced variations concentrate on questions that were already close calls. Ambiguity is very divergent, becoming disagreement.
The scoring harness introduces a disjoint layer column. In the same generations, LM eval's strict filter disagreed with our extractor on 9/1,319 cases. All nine were correct answers formatted as "n.00". None in the fragile set. Disagreement between two standard harnesses on the same generation is comparable to deployment-induced verdict noise.
Config-to-config deltas are re-rolls plus a small stable core
Between BF16 batch caps, cross-cell flip rates (0.88–1.02%) sit at the within-cell floor, and 100% of the differing questions lie inside the fragile set. Comparing batch caps here measures deployment noise, not batch-cap quality.
Between arms it is 65–83% re-roll, and the remainder is structure: the same 15 questions separate FP8 from every BF16 cell, the same 11 for INT4, none ever flipping within a cell, with directions that nearly cancel (FP8 correct on 8 of 15, INT4 on 8 of 11). Three arms sit within 0.13 accuracy points while giving deterministically different answers on 11–15 questions each.
The backend axis tells the same story: HF 95.15% vs vLLM 95.22% (one net question), but 15 verdict flips and 0/1,319 bitwise-identical outputs. Fourteen are fragile-set re-rolls. The fifteenth, q539, is the cleanest stable structure in the study: all 14 vLLM BF16 runs answer 30; HF and both quantized arms answer 35, the gold.
Within the variance we measured: small accuracy differences between configs of the same model are mostly re-rolls of a concentrated fragile set, and near the ceiling the sign of a small delta carries little information.
Figure 3. Verdict map over 23 runs: the 55 fragile questions, and the 22 stable arm disagreements in three signature groups.
What we think is happening
At some token positions the top-two logits are nearly tied because the question is nearly tied, and deployment-dependent kernel choices perturb logits by a small δ that swaps the argmax only there. The pattern fits (early near-uniform divergence, same final answers, flips concentrated in ambiguous questions, zero churn where per-deployment kernel selection is absent), but we did not instrument logits; this is a hypothesis. The direct test is cheap: top-two logit gaps at first-divergence positions.
What to do differently. (1) Pin compiled/autotuned artifacts like weights and seeds when exact reproduction matters: one shared cache froze our BF16 path bitwise. (2) Estimate variance by repeating across fresh deployments; repeats inside one engine measure zero here. (3) Report per-question churn next to accuracy: a 0.8–1.3% per-pair flip floor leaves smaller single-run deltas unresolved. (4) Publish raw per-sample outputs: 9/1,319 verdicts changed with the scoring harness alone.
Limitations
One model, one benchmark, one engine version, one GPU family and driver: the protocol transfers; the rates are point estimates. The loader↔loader result is one pair from one cache build and needs replication. With 3–5 repeats the 55-question fragile set is likely an undercount. Cap 32's higher output identity (64% vs 59%) is unexplained. HF repeatability was measured in-process only. FP8 means vLLM's online dynamic W8A8 on H100. Thinking mode and sampled decoding are out of scope.
Where this sits
Thinking Machines traced served-LLM nondeterminism to batch composition under load; we confirm determinism at fixed composition on a GDN hybrid, then show the operative cross-deployment variable is compile state, which is constant in their single-process setting.
The Silent Hyperparameter holds weights, decoding, and hardware fixed while varying the engine and finds shifts up to 16.6 points; we fix one engine and show that even one pinned implementation is not one measurement.
The lesson
The recipe was never wrong, only incomplete. Weights, seed, and sampler pin the function after the deployment pins the kernels: here the residual was compile/autotune state, and one shipped cache file closed it to the last token. An accuracy number is a fine summary and a poor microscope. When reproducibility matters, specify the deployment or repeat over it, and keep the per-question records that tell noise from structure.
Reproduce it.
bashCopy code
Pins: Qwen/Qwen3.5-9B @ c2022362; Intel/Qwen3.5-9B-int4-AutoRound @ 29688b89; GSM8K parquet @ 740312a; vLLM 0.25.0 on H100-80GB (full pins in the repo). Every run's per-question record (raw text, token IDs, rendered prompt, extraction path, verdict) is committed, so everything re-scores. Cost: ≈$85–90.
Disputable golds (annotated, not excluded). q423: gold 8, literal arithmetic −8. q510: the gold's day count is one of three defensible readings.