# LLM-42: Enabling Determinism in LLM Inference with Verified Speculation

**Authors:** Raja Gond, Aditya K Kamath, Ramachandran Ramjee, Ashish Panwar  
**arXiv:** [2601.17768](https://arxiv.org/abs/2601.17768)  
**Date:** 2026-01-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2601.17768v2/

## Key points

- Achieving determinism in Large Language Model (LLM) inference has been a challenge due to floating-point non-associativity, dynamic batching, and varying reduction orders in GPU kernels.
- Disabling dynamic batching can eliminate non-determinism but degrades throughput.
- LLM-42 introduces a scheduling-based approach inspired by speculative decoding techniques to enable determinism in LLM inference.
- LLM-42 leverages shape-consistent reductions and decodes tokens using a non-deterministic fast path while enforcing determinism through a lightweight verify-rollback loop.
- The verifier in LLM-42 replays candidate tokens under a fixed-shape reduction schedule to ensure consistency across runs.
- LLM-42 mostly reuses existing kernels without significant modifications and incurs overhead proportional to the portion of traffic requiring determinism.
- Table 4 presents key metrics characterizing throughput in offline inference for both SGLang and LLM-42, showcasing the total number of rollbacks for different datasets/configurations under each model.

## Summary

In the field of Large Language Model (LLM) inference, achieving determinism in output across different runs has been a challenge due to floating-point non-associativity, dynamic batching, and varying reduction orders in GPU kernels. While disabling dynamic batching can eliminate non-determinism, it comes at a cost of degrading throughput. Another approach involves making kernels batch-invariant; however, this requires new implementations and imposes fixed runtime overheads. Inspired by speculative decoding techniques, a novel scheduling-based approach called LLM-42 has been introduced to enable determinism in LLM inference. The key insight behind LLM-42 is that if a sequence is in a consistent state, the next emitted token is likely to be consistent even with dynamic batching. Additionally, most GPU kernels utilize shape-consistent reductions. Leveraging these observations, LLM-42 decodes tokens using a non-deterministic fast path and enforces determinism through a lightweight verify-rollback loop. The verifier in LLM-42 replays candidate tokens under a fixed-shape reduction schedule: committing those that are guaranteed to be consistent across runs and rolling back those that violate determinism. Notably, LLM-42 mostly reuses existing kernels without significant modifications and only incurs overhead proportional to the portion of traffic requiring determinism. Table 4 presents two key metrics characterizing throughput in offline inference for both SGLang and LLM-42. SGLang operates in deterministic or non-deterministic mode for all requests while LLM-42 supports selective determinism based on the fraction of traffic requiring deterministic output. The table showcases the total number of rollbacks for different datasets/configurations under each model. In conclusion, By leveraging insights from speculative decoding and shape-consistent reductions, LLM-42 provides a practical method for ensuring consistency in output across multiple runs without imposing significant runtime overheads or requiring extensive kernel redesigns.