AI summary · arXiv: 2611.12832v1 · cs.LG
What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
The essentials
Emulated NVFP4 pretraining reaches similar validation loss to BF16 while requiring more exposures to acquire rare facts, with gradient noise through Adam explaining part of the cost.
Question
Does matching validation loss mean that low-precision pretraining acquires rare facts as efficiently as BF16? The paper also investigates how gradient error contributes to any difference in exposure efficiency.
Method
The authors trained 12M, 40M, and 100M transformers with BF16 and emulated FP8, MXFP4, or NVFP4 matrix multiplications on synthetic biographies with controlled exposure counts, and trained 40M models on biographies mixed into a 1B-token FineWeb-Edu stream. They measured recall across exposure levels and defined an exposure multiplier relative to BF16; primary comparisons used six seeds. To study mechanisms, they separately quantized passes, injected gradient noise, split noise between Adam’s moments, and evaluated a checkpoint-based prediction across 17 configurations.
Findings
-
Result
At matched tokens, NVFP4 required 1.32 times as many exposures as BF16 on synthetic data and 1.52 times as many on natural text, despite small validation-loss differences.
See the passage in the paper
On synthetic data (40M) NVFP4 shifts the curve right by 1.32$\times$ [1.16, 1.50] at matched tokens and 1.25$\times$ [1.13, 1.38] at matched validation loss, with every seed above 1 (sign test $p=0.03$), while the final validation loss differs by only 0.1–0.9%.
-
Result
Backward-only quantization increased exposure costs while changing validation loss by only 0.1% on average, and zero-mean gradient noise reproduced the cost.
See the passage in the paper
Backward-only quantization runs the forward pass in BF16 and changes validation loss by just $+0.1\%$ on average, so its cost is nearly invisible in the loss. Its gradient error is approximately zero-mean (stochastic rounding, Appendix D), and exactly zero-mean noise reproduces its cost.
-
Result
In noise controls at twice the NVFP4 error, noise in Adam’s update accounted for about two thirds of the measured log-cost and persisted through training, whereas the second-moment-only cost faded.
See the passage in the paper
At equal expected step, noise in both moments costs 1.21$\times$ [1.05, 1.39] more than noise in the second moment only (paired, $p=0.017$, 7 of 8 seeds), two thirds of its $\log$-cost of 1.33$\times$ (seed bootstrap 38–91%).
-
Result
The checkpoint-based diagnostic ranked exposure costs across 17 tested configurations with Spearman correlation 0.96, but did not calibrate every cost accurately.
See the passage in the paper
Across 17 configurations (formats, partial quantization, and i.i.d., block-shaped and forward noise at three scales each), the prediction ranks the observed costs with Spearman correlation 0.96 (bootstrap interval over configurations [0.84, 0.99]).
-
Authors’ interpretation
The authors argue that low-precision recipes should be assessed with exposure-controlled factual recall alongside validation loss.
See the passage in the paper
We therefore suggest reporting an exposure-controlled recall curve alongside the loss gap.
Limitations
- The experiments use models up to 96.6M parameters and 1B tokens, and the authors leave generalization to production scale unresolved.
See the passage in the paper
Our models have at most 96.6M parameters and see at most 1B tokens. Production-scale generalization remains open, and the natural-text setting reaches about 24 tokens per parameter.
- The FP4 operations were emulated, without validation against native hardware kernels or measurement of native throughput.
See the passage in the paper
We do not validate bitwise agreement with Blackwell kernels or measure native FP4 throughput.
- The factual probes use synthetic biographies and training-seen templates, and the study does not test paraphrased questions.
See the passage in the paper
Facts are synthetic biographies, on their own or mixed into one web corpus, rather than naturally occurring long-tail knowledge. We test only AdamW, a three-rate sweep at 40M with three seeds, and one alternative schedule.
Why it matters
The authors argue that validation loss and aggregate benchmarks can conceal slower acquisition of rare facts during low-precision training. They propose adding exposure-controlled factual recall to evaluations of low-precision recipes, while noting that its generalization beyond their tested models remains untested.
Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Among the 23 papers around this one, the name that comes back most: Brian Chmiel (3 papers)
3 more people sign several of them. The full map adds their institutions and approaches, from the papers of the last years.
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Training LLMs with MXFP4Albert Tseng, Tao Yu, Youngsuk Park · 2025 · International Conference on Artificial Intelligence and Statistics · cited 50 times · Not summarized yet
Training LLMs with MXFP4 proposes stochastic rounding and Hadamard transforms for near-lossless MXFP4 GEMMs, while this paper measures rare-fact exposure costs under low precision.
Abstract
Low precision (LP) datatypes such as MXFP4 can accelerate matrix multiplications (GEMMs) and reduce training costs. However, directly using MXFP4 instead of BF16 during training significantly degrades model quality. In this work, we present the first near-lossless training recipe that uses MXFP4 GEMMs, which are $2\times$ faster than FP8 on supported hardware. Our key insight is to compute unbiased gradient estimates with stochastic rounding (SR), resulting in more accurate model updates. However, directly applying SR to MXFP4 can result in high variance from block-level outliers, harming convergence. To overcome this, we use the random Hadamard tranform to theoretically bound the variance o…
-
FP4 All the Way: Fully Quantized Training of LLMsBrian Chmiel, Maxim Fishman, Ron Banner et al. · 2025 · Machine Learning · preprint, not peer reviewed · cited 48 times · AI summary available
FP4 All the Way demonstrates fully quantized LLM training with format and rounding choices, while this paper evaluates how FP4 changes rare-fact acquisition.
Abstract
We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets up to 200 billion tokens. We extensively investigate key design choices for FP4, including block sizes, scaling formats, and rounding methods. Our analysis shows that the NVFP4 format, where each block of 16 FP4 values (E2M1) shares a scale represented in E4M3, provides optimal results. We use stochastic rounding for backward and update passes and round-to-nearest for the forward pass to enhance stability. Additionally, we identify a theoretical and empirical threshold for effective quan…
-
Pretraining Large Language Models with NVFP4Nvidia, Felix Abecassis, A. Agrusa et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 104 times · Not summarized yet
Pretraining Large Language Models with NVFP4 studies efficient NVFP4 pretraining, while this paper measures its rare-fact exposure efficiency against BF16.
Abstract
Large Language Models (LLMs) today are powerful problem solvers across many domains, and they continue to get stronger as they scale in model size, training set size, and training set quality, as shown by extensive research and experimentation across the industry. Training a frontier model today requires on the order of tens to hundreds of yottaflops, which is a massive investment of time, compute, and energy. Improving pretraining efficiency is therefore essential to enable the next generation of even more capable LLMs. While 8-bit floating point (FP8) training is now widely adopted, transitioning to even narrower precision, such as 4-bit floating point (FP4), could unlock additional improv…
-
Linger and Lose: Knowledge Collapse in Low-Bit Language ModelsPrashanna Mani Paudel, S. Sheshappanavar · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet
Linger and Lose measures hidden knowledge loss under low-bit training on synthetic biographies, while this paper studies rare-fact acquisition under FP4.
Abstract
Training language models with ternary weights is commonly judged by loss and downstream accuracy, which record only a modest cost relative to full precision. We show that these metrics can conceal a much larger failure. We instead measure knowledge capacity, the factual bits stored per parameter, on synthetic biographies with known information content. We train GPT-2-style models from scratch with 2.5M to 50M parameters at five precisions. Under the standard cosine schedule, ternary models retain as little as 6% of an identically trained fp16 model's capacity. The deficit widens with model size while perplexity rises by only 1.4 to 1.6 times. Measured throughout training, these models first …
-
Accurate Neural Training with 4-bit Matrix Multiplications at Standard FormatsBrian Chmiel, Ron Banner, Elad Hoffer et al. · 2021 · International Conference on Learning Representations · cited 41 times · Not summarized yet
Accurate Neural Training with 4-bit Matrix Multiplications combines unbiased, logarithmic gradient quantization, an approach relevant to the paper’s study of gradient error.
Abstract
Quantization of the weights and activations is one of the main methods to reduce the computational footprint of Deep Neural Networks (DNNs) training. Current methods enable 4-bit quantization of the forward phase. However, this constitutes only a third of the training process. Reducing the computational footprint of the entire training process requires the quantization of the neural gradients, i.e., the loss gradients with respect to the outputs of intermediate neural layers. Previous works separately showed that accurate 4-bit quantization of the neural gradients needs to (1) be unbiased and (2) have a log scale. However, no previous work aimed to combine both ideas, as we do in this work. …
-
Scaling Laws for PrecisionTanishq Kumar, Zachary Ankner, Benjamin F. Spector et al. · 2024 · Machine Learning · cited 116 times · AI summary available
Scaling Laws for Precision predicts low-precision training loss through effective parameter counts, while this paper tests rare-fact recall at matched loss.
Abstract
Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware" scaling laws for both training and inference. We propose that training in lower precision reduces the model's "effective parameter count," allowing us to predict the additional loss incurred from training in low precision and post-train quantization. For inference, we find that the degradation introduced by post-training quantization increases as models are trained on more data, eventually making additional pretraining data actively harmful. For training, our scaling laws allow us to predict the loss of a model w…
-
The Unreasonable Ineffectiveness of the Deeper LayersAndrey Gromov, Kushal Tirumala, Hassan Shapourian et al. · 2024 · Computation and Language · cited 302 times · AI summary available
The Unreasonable Ineffectiveness of the Deeper Layers studies layer pruning and question-answering performance, rather than precision’s effect on pretraining facts.
Abstract
We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed. To prune these models, we identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. In particular, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single A100 GPU. From a practical perspective, these results suggest that l…
-
The Curse and Blessing of Mean Bias in FP4-Quantized LLM TrainingHeng-Jie Cao, Zhen-Dong Huang, Meng-Yi Chen et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet
The Curse and Blessing of Mean Bias investigates why FP4 training loses quality, while this paper examines FP4’s costs for acquiring rare facts.
Abstract
FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitudes, which inflate dynamic range and compress long-tail signals. We identify a counterintuitive source of this failure: dominant activation outliers are not merely arbitrary sparse events, but are largely induced by a coherent rank-one mean bias, whose direction aligns with the leading anisotropic spectral component. This mean component strengthens during training, is amplified and reshaped by attention and FFN operators, and increasingly dominates top activation magnitudes. Crucially, this discovery reveals that a…
Prior work it builds on
-
Optimizing Large Language Model Training Using FP4 QuantizationRuizhe Wang, Yeyun Gong, Xiao Liu et al. · 2025 · International Conference on Machine Learning · cited 74 times · Not summarized yet
Optimizing Large Language Model Training Using FP4 Quantization proposes estimators and outlier compensation for FP4 training, while this paper measures rare-fact exposure costs.
Abstract
The growing computational demands of training large language models (LLMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantization errors and limited representational capacity. This work introduces the first FP4 training framework for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an outlier clamping and compensation strategy to prevent activation collapse. To ensure stability, the framework integrates…
-
FP4 All the Way: Fully Quantized Training of Large Language ModelsBrian Chmiel, Maxim Fishman, Ron Banner et al. · 2025 · Neural Information Processing Systems · cited 13 times · Not summarized yet
FP4 All the Way presents fully quantized LLM training, whereas this paper evaluates rare-fact acquisition under low-precision pretraining.
-
Deep Learning with Limited Numerical PrecisionSuyog Gupta, Ankur Agrawal, K. Gopalakrishnan et al. · 2015 · International Conference on Machine Learning · cited 2,339 times · Not summarized yet
Deep Learning with Limited Numerical Precision studies rounding effects in low-precision network training, relevant to this paper’s analysis of gradient noise.
Abstract
Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithm…
-
Mixed Precision TrainingP. Micikevicius, Sharan Narang, Jonah Alben et al. · 2017 · International Conference on Learning Representations · cited 2,736 times · Not summarized yet
Mixed Precision Training introduces techniques for training neural networks with reduced precision, providing background for the paper’s low-precision comparisons.
Abstract
Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using half precision floating point numbers. In our technique, weights, activations and gradients are stored in IEEE half-precision format. Half-precision floating numbers have limited numerical range compared to single-precision numbers. We propose two techniques to handle this loss of information. Firstly, we recommend maintaining a single-precision copy of the weights that accumulate…
-
How Do Large Language Models Acquire Factual Knowledge During Pretraining?Hoyeon Chang, Jinho Park, Seonghyeon Ye et al. · 2024 · Computation and Language · cited 134 times · AI summary available
How Do Large Language Models Acquire Factual Knowledge During Pretraining studies factual acquisition and forgetting, while this paper measures rare-fact recall across exposure levels.
Abstract
Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model's capability to acquire and maintain factual knowledge. Next, there is a power-law relationship between training steps and forgetting of memorization and ge…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 16 November 2026.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.