What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts

Authors: Chang Lu, Mengjiao Zhang

Submitted to arXiv on 09 Nov. 2026
Abstract

Low-precision pretraining is usually judged by validation loss. We ask whether matching loss also means learning rare facts as efficiently. We train transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, on their own and mixed into web text. NVFP4 needs 1.32$\times$ and 1.52$\times$ as many exposures as BF16 to match its recall while validation loss differs by only 0.5% and 2%, and 1.26$\times$ and 1.30$\times$ when both models are evaluated with BF16 matrix multiplications. Under our primary readout, the exposure cost is about 1.2$\times$ at matched loss. Zero-mean gradient noise reproduces the cost of quantizing only the backward pass, which barely moves the loss. In a controlled experiment at twice the NVFP4 error, noise fed only to Adam's second-moment estimate acts like a lower learning rate and its cost fades, whereas noise in the update lowers its signal-to-noise ratio and carries about two thirds of the cost through the end of training. Raising the learning rate to offset the smaller average step does not recover the cost, either for this noise or for NVFP4. Gradient statistics of BF16 checkpoints rank the costs of formats, partial quantizations and noise controls without fitting (Spearman 0.96). We suggest measuring rare-fact acquisition alongside loss when evaluating low-precision recipes.

The papers around it, by date and closeness

Higher is closer to this paper; left is older. Lines join papers close to each other: groups of lines are sub-topics. Select a point to see why it is there. Back to the summary of this paper

How to read this map? Illustrated guide

Closest papers Prior work it builds on Papers that cite it

Papers around this one

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Close Another approach to the problem Cited by this paper
    Training LLMs with MXFP4
    Albert Tseng, Tao Yu, Youngsuk Park · 2025 · International Conference on Artificial Intelligence and Statistics · cited 50 times · Not summarized yet

    Training LLMs with MXFP4 proposes stochastic rounding and Hadamard transforms for near-lossless MXFP4 GEMMs, while this paper measures rare-fact exposure costs under low precision.

    Abstract

    Low precision (LP) datatypes such as MXFP4 can accelerate matrix multiplications (GEMMs) and reduce training costs. However, directly using MXFP4 instead of BF16 during training significantly degrades model quality. In this work, we present the first near-lossless training recipe that uses MXFP4 GEMMs, which are $2\times$ faster than FP8 on supported hardware. Our key insight is to compute unbiased gradient estimates with stochastic rounding (SR), resulting in more accurate model updates. However, directly applying SR to MXFP4 can result in high variance from block-level outliers, harming convergence. To overcome this, we use the random Hadamard tranform to theoretically bound the variance o…

  2. Close Another approach to the problem
    FP4 All the Way: Fully Quantized Training of LLMs
    Brian Chmiel, Maxim Fishman, Ron Banner et al. · 2025 · Machine Learning · preprint, not peer reviewed · cited 48 times · AI summary available

    FP4 All the Way demonstrates fully quantized LLM training with format and rounding choices, while this paper evaluates how FP4 changes rare-fact acquisition.

    Abstract

    We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets up to 200 billion tokens. We extensively investigate key design choices for FP4, including block sizes, scaling formats, and rounding methods. Our analysis shows that the NVFP4 format, where each block of 16 FP4 values (E2M1) shares a scale represented in E4M3, provides optimal results. We use stochastic rounding for backward and update passes and round-to-nearest for the forward pass to enhance stability. Additionally, we identify a theoretical and empirical threshold for effective quan…

  3. Close Another approach to the problem Cited by this paper
    Pretraining Large Language Models with NVFP4
    Nvidia, Felix Abecassis, A. Agrusa et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 104 times · Not summarized yet

    Pretraining Large Language Models with NVFP4 studies efficient NVFP4 pretraining, while this paper measures its rare-fact exposure efficiency against BF16.

    Abstract

    Large Language Models (LLMs) today are powerful problem solvers across many domains, and they continue to get stronger as they scale in model size, training set size, and training set quality, as shown by extensive research and experimentation across the industry. Training a frontier model today requires on the order of tens to hundreds of yottaflops, which is a massive investment of time, compute, and energy. Improving pretraining efficiency is therefore essential to enable the next generation of even more capable LLMs. While 8-bit floating point (FP8) training is now widely adopted, transitioning to even narrower precision, such as 4-bit floating point (FP4), could unlock additional improv…

  4. Close Same question Cited by this paper
    Linger and Lose: Knowledge Collapse in Low-Bit Language Models
    Prashanna Mani Paudel, S. Sheshappanavar · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

    Linger and Lose measures hidden knowledge loss under low-bit training on synthetic biographies, while this paper studies rare-fact acquisition under FP4.

    Abstract

    Training language models with ternary weights is commonly judged by loss and downstream accuracy, which record only a modest cost relative to full precision. We show that these metrics can conceal a much larger failure. We instead measure knowledge capacity, the factual bits stored per parameter, on synthetic biographies with known information content. We train GPT-2-style models from scratch with 2.5M to 50M parameters at five precisions. Under the standard cosine schedule, ternary models retain as little as 6% of an identically trained fp16 model's capacity. The deficit widens with model size while perplexity rises by only 1.4 to 1.6 times. Measured throughout training, these models first …

  5. Related Groundwork for this paper Cited by this paper
    Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats
    Brian Chmiel, Ron Banner, Elad Hoffer et al. · 2021 · International Conference on Learning Representations · cited 41 times · Not summarized yet

    Accurate Neural Training with 4-bit Matrix Multiplications combines unbiased, logarithmic gradient quantization, an approach relevant to the paper’s study of gradient error.

    Abstract

    Quantization of the weights and activations is one of the main methods to reduce the computational footprint of Deep Neural Networks (DNNs) training. Current methods enable 4-bit quantization of the forward phase. However, this constitutes only a third of the training process. Reducing the computational footprint of the entire training process requires the quantization of the neural gradients, i.e., the loss gradients with respect to the outputs of intermediate neural layers. Previous works separately showed that accurate 4-bit quantization of the neural gradients needs to (1) be unbiased and (2) have a log scale. However, no previous work aimed to combine both ideas, as we do in this work. …

  6. Related Another approach to the problem Cited by this paper
    Scaling Laws for Precision
    Tanishq Kumar, Zachary Ankner, Benjamin F. Spector et al. · 2024 · Machine Learning · cited 116 times · AI summary available

    Scaling Laws for Precision predicts low-precision training loss through effective parameter counts, while this paper tests rare-fact recall at matched loss.

    Abstract

    Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware" scaling laws for both training and inference. We propose that training in lower precision reduces the model's "effective parameter count," allowing us to predict the additional loss incurred from training in low precision and post-train quantization. For inference, we find that the degradation introduced by post-training quantization increases as models are trained on more data, eventually making additional pretraining data actively harmful. For training, our scaling laws allow us to predict the loss of a model w…

  7. Related Broader context
    The Unreasonable Ineffectiveness of the Deeper Layers
    Andrey Gromov, Kushal Tirumala, Hassan Shapourian et al. · 2024 · Computation and Language · cited 302 times · AI summary available

    The Unreasonable Ineffectiveness of the Deeper Layers studies layer pruning and question-answering performance, rather than precision’s effect on pretraining facts.

    Abstract

    We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed. To prune these models, we identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. In particular, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single A100 GPU. From a practical perspective, these results suggest that l…

  8. Related Same question Cited by this paper
    The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training
    Heng-Jie Cao, Zhen-Dong Huang, Meng-Yi Chen et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

    The Curse and Blessing of Mean Bias investigates why FP4 training loses quality, while this paper examines FP4’s costs for acquiring rare facts.

    Abstract

    FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitudes, which inflate dynamic range and compress long-tail signals. We identify a counterintuitive source of this failure: dominant activation outliers are not merely arbitrary sparse events, but are largely induced by a coherent rank-one mean bias, whose direction aligns with the leading anisotropic spectral component. This mean component strengthens during training, is amplified and reshaped by attention and FFN operators, and increasingly dominates top activation magnitudes. Crucially, this discovery reveals that a…

Prior work it builds on

  1. Related Another approach to the problem
    Optimizing Large Language Model Training Using FP4 Quantization
    Ruizhe Wang, Yeyun Gong, Xiao Liu et al. · 2025 · International Conference on Machine Learning · cited 74 times · Not summarized yet

    Optimizing Large Language Model Training Using FP4 Quantization proposes estimators and outlier compensation for FP4 training, while this paper measures rare-fact exposure costs.

    Abstract

    The growing computational demands of training large language models (LLMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantization errors and limited representational capacity. This work introduces the first FP4 training framework for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an outlier clamping and compensation strategy to prevent activation collapse. To ensure stability, the framework integrates…

  2. Related* Another approach to the problem Used in depth by this paper
    FP4 All the Way: Fully Quantized Training of Large Language Models
    Brian Chmiel, Maxim Fishman, Ron Banner et al. · 2025 · Neural Information Processing Systems · cited 13 times · Not summarized yet

    FP4 All the Way presents fully quantized LLM training, whereas this paper evaluates rare-fact acquisition under low-precision pretraining.

  3. Related Groundwork for this paper
    Deep Learning with Limited Numerical Precision
    Suyog Gupta, Ankur Agrawal, K. Gopalakrishnan et al. · 2015 · International Conference on Machine Learning · cited 2,339 times · Not summarized yet

    Deep Learning with Limited Numerical Precision studies rounding effects in low-precision network training, relevant to this paper’s analysis of gradient noise.

    Abstract

    Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithm…

  4. Related Groundwork for this paper
    Mixed Precision Training
    P. Micikevicius, Sharan Narang, Jonah Alben et al. · 2017 · International Conference on Learning Representations · cited 2,736 times · Not summarized yet

    Mixed Precision Training introduces techniques for training neural networks with reduced precision, providing background for the paper’s low-precision comparisons.

    Abstract

    Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using half precision floating point numbers. In our technique, weights, activations and gradients are stored in IEEE half-precision format. Half-precision floating numbers have limited numerical range compared to single-precision numbers. We propose two techniques to handle this loss of information. Firstly, we recommend maintaining a single-precision copy of the weights that accumulate…

  5. Related Same question
    How Do Large Language Models Acquire Factual Knowledge During Pretraining?
    Hoyeon Chang, Jinho Park, Seonghyeon Ye et al. · 2024 · Computation and Language · cited 134 times · AI summary available

    How Do Large Language Models Acquire Factual Knowledge During Pretraining studies factual acquisition and forgetting, while this paper measures rare-fact recall across exposure levels.

    Abstract

    Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model's capability to acquire and maintain factual knowledge. Next, there is a power-law relationship between training steps and forgetting of memorization and ge…

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 16 November 2026.

The illustrated guide to the papers around a paper