What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts

Auteurs : Chang Lu, Mengjiao Zhang

Soumis à arXiv le 09 Nov. 2026
Résumé scientifique

Low-precision pretraining is usually judged by validation loss. We ask whether matching loss also means learning rare facts as efficiently. We train transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, on their own and mixed into web text. NVFP4 needs 1.32$\times$ and 1.52$\times$ as many exposures as BF16 to match its recall while validation loss differs by only 0.5% and 2%, and 1.26$\times$ and 1.30$\times$ when both models are evaluated with BF16 matrix multiplications. Under our primary readout, the exposure cost is about 1.2$\times$ at matched loss. Zero-mean gradient noise reproduces the cost of quantizing only the backward pass, which barely moves the loss. In a controlled experiment at twice the NVFP4 error, noise fed only to Adam's second-moment estimate acts like a lower learning rate and its cost fades, whereas noise in the update lowers its signal-to-noise ratio and carries about two thirds of the cost through the end of training. Raising the learning rate to offset the smaller average step does not recover the cost, either for this noise or for NVFP4. Gradient statistics of BF16 checkpoints rank the costs of formats, partial quantizations and noise controls without fitting (Spearman 0.96). We suggest measuring rare-fact acquisition alongside loss when evaluating low-precision recipes.

Les articles autour de lui, par date et par proximité

Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article

Comment lire cette carte ? Guide illustré

Les articles les plus proches Les travaux antérieurs sur lesquels il s’appuie Les articles qui le citent

Les articles autour de celui-ci

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

  1. Proche Autre approche du problème Cité par cet article
    Training LLMs with MXFP4
    Albert Tseng, Tao Yu, Youngsuk Park · 2025 · International Conference on Artificial Intelligence and Statistics · cité 50 fois · Pas encore résumé

    Training LLMs with MXFP4 propose des GEMM MXFP4 et un arrondi stochastique pour préserver la qualité, tandis que l’article mesure le coût sur l’acquisition des faits rares.

    Résumé scientifique

    Low precision (LP) datatypes such as MXFP4 can accelerate matrix multiplications (GEMMs) and reduce training costs. However, directly using MXFP4 instead of BF16 during training significantly degrades model quality. In this work, we present the first near-lossless training recipe that uses MXFP4 GEMMs, which are $2\times$ faster than FP8 on supported hardware. Our key insight is to compute unbiased gradient estimates with stochastic rounding (SR), resulting in more accurate model updates. However, directly applying SR to MXFP4 can result in high variance from block-level outliers, harming convergence. To overcome this, we use the random Hadamard tranform to theoretically bound the variance o…

  2. Proche Autre approche du problème
    FP4 All the Way: Fully Quantized Training of LLMs
    Brian Chmiel, Maxim Fishman, Ron Banner et al. · 2025 · Machine Learning · preprint, non relu par les pairs · cité 48 fois · Résumé IA disponible

    FP4 All the Way étudie l’entraînement entièrement quantifié des LLM en FP4 et ses choix de format et d’arrondi, plutôt que le rappel des faits rares.

    Résumé scientifique

    We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets up to 200 billion tokens. We extensively investigate key design choices for FP4, including block sizes, scaling formats, and rounding methods. Our analysis shows that the NVFP4 format, where each block of 16 FP4 values (E2M1) shares a scale represented in E4M3, provides optimal results. We use stochastic rounding for backward and update passes and round-to-nearest for the forward pass to enhance stability. Additionally, we identify a theoretical and empirical threshold for effective quan…

  3. Proche Autre approche du problème Cité par cet article
    Pretraining Large Language Models with NVFP4
    Nvidia, Felix Abecassis, A. Agrusa et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 104 fois · Pas encore résumé

    Pretraining Large Language Models with NVFP4 cherche à améliorer l’efficacité du préentraînement en NVFP4, tandis que l’article évalue son effet sur l’acquisition de faits rares.

    Résumé scientifique

    Large Language Models (LLMs) today are powerful problem solvers across many domains, and they continue to get stronger as they scale in model size, training set size, and training set quality, as shown by extensive research and experimentation across the industry. Training a frontier model today requires on the order of tens to hundreds of yottaflops, which is a massive investment of time, compute, and energy. Improving pretraining efficiency is therefore essential to enable the next generation of even more capable LLMs. While 8-bit floating point (FP8) training is now widely adopted, transitioning to even narrower precision, such as 4-bit floating point (FP4), could unlock additional improv…

  4. Proche Même question Cité par cet article
    Linger and Lose: Knowledge Collapse in Low-Bit Language Models
    Prashanna Mani Paudel, S. Sheshappanavar · 2026 · arXiv.org · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé

    Linger and Lose mesure les connaissances factuelles acquises par des modèles basse précision sur des biographies synthétiques et montre que perte et perplexité masquent leur déficit.

    Résumé scientifique

    Training language models with ternary weights is commonly judged by loss and downstream accuracy, which record only a modest cost relative to full precision. We show that these metrics can conceal a much larger failure. We instead measure knowledge capacity, the factual bits stored per parameter, on synthetic biographies with known information content. We train GPT-2-style models from scratch with 2.5M to 50M parameters at five precisions. Under the standard cosine schedule, ternary models retain as little as 6% of an identically trained fp16 model's capacity. The deficit widens with model size while perplexity rises by only 1.4 to 1.6 times. Measured throughout training, these models first …

  5. Apparenté Autre approche du problème Cité par cet article
    Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats
    Brian Chmiel, Ron Banner, Elad Hoffer et al. · 2021 · International Conference on Learning Representations · cité 41 fois · Pas encore résumé

    Accurate Neural Training combine quantification 4 bits non biaisée et échelle logarithmique pour les gradients, alors que l’article étudie les faits rares sous calcul FP4.

    Résumé scientifique

    Quantization of the weights and activations is one of the main methods to reduce the computational footprint of Deep Neural Networks (DNNs) training. Current methods enable 4-bit quantization of the forward phase. However, this constitutes only a third of the training process. Reducing the computational footprint of the entire training process requires the quantization of the neural gradients, i.e., the loss gradients with respect to the outputs of intermediate neural layers. Previous works separately showed that accurate 4-bit quantization of the neural gradients needs to (1) be unbiased and (2) have a log scale. However, no previous work aimed to combine both ideas, as we do in this work. …

  6. Apparenté Contexte plus large Cité par cet article
    Scaling Laws for Precision
    Tanishq Kumar, Zachary Ankner, Benjamin F. Spector et al. · 2024 · Machine Learning · cité 116 fois · Résumé IA disponible

    Scaling Laws for Precision modélise les pertes dues à la faible précision en entraînement et en inférence, sans mesurer l’acquisition de faits rares.

    Résumé scientifique

    Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware" scaling laws for both training and inference. We propose that training in lower precision reduces the model's "effective parameter count," allowing us to predict the additional loss incurred from training in low precision and post-train quantization. For inference, we find that the degradation introduced by post-training quantization increases as models are trained on more data, eventually making additional pretraining data actively harmful. For training, our scaling laws allow us to predict the loss of a model w…

  7. Apparenté Contexte plus large
    The Unreasonable Ineffectiveness of the Deeper Layers
    Andrey Gromov, Kushal Tirumala, Hassan Shapourian et al. · 2024 · Computation and Language · cité 302 fois · Résumé IA disponible

    The Unreasonable Ineffectiveness of the Deeper Layers étudie l’élagage et le réglage fin de couches de LLM, sans aborder la précision d’entraînement ni les faits rares.

    Résumé scientifique

    We empirically study a simple layer-pruning strategy for popular families of open-weight pretrained LLMs, finding minimal degradation of performance on different question-answering benchmarks until after a large fraction (up to half) of the layers are removed. To prune these models, we identify the optimal block of layers to prune by considering similarity across layers; then, to "heal" the damage, we perform a small amount of finetuning. In particular, we use parameter-efficient finetuning (PEFT) methods, specifically quantization and Low Rank Adapters (QLoRA), such that each of our experiments can be performed on a single A100 GPU. From a practical perspective, these results suggest that l…

  8. Apparenté Autre approche du problème Cité par cet article
    The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training
    Heng-Jie Cao, Zhen-Dong Huang, Meng-Yi Chen et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 2 fois · Pas encore résumé

    The Curse and Blessing of Mean Bias analyse les outliers d’activations qui fragilisent l’entraînement FP4, tandis que l’article relie le bruit FP4 au rappel des faits rares.

    Résumé scientifique

    FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitudes, which inflate dynamic range and compress long-tail signals. We identify a counterintuitive source of this failure: dominant activation outliers are not merely arbitrary sparse events, but are largely induced by a coherent rank-one mean bias, whose direction aligns with the leading anisotropic spectral component. This mean component strengthens during training, is amplified and reshaped by attention and FFN operators, and increasingly dominates top activation magnitudes. Crucially, this discovery reveals that a…

Les travaux antérieurs sur lesquels il s’appuie

  1. Apparenté Autre approche du problème
    Optimizing Large Language Model Training Using FP4 Quantization
    Ruizhe Wang, Yeyun Gong, Xiao Liu et al. · 2025 · International Conference on Machine Learning · cité 74 fois · Pas encore résumé

    Optimizing Large Language Model Training Using FP4 Quantization propose un cadre d’entraînement FP4 pour les LLM, tandis que l’article examine l’acquisition de faits rares sous quantification.

    Résumé scientifique

    The growing computational demands of training large language models (LLMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantization errors and limited representational capacity. This work introduces the first FP4 training framework for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an outlier clamping and compensation strategy to prevent activation collapse. To ensure stability, the framework integrates…

  2. Apparenté* Autre approche du problème Utilisé en profondeur par cet article
    FP4 All the Way: Fully Quantized Training of Large Language Models
    Brian Chmiel, Maxim Fishman, Ron Banner et al. · 2025 · Neural Information Processing Systems · cité 13 fois · Pas encore résumé

    FP4 All the Way présente l’entraînement entièrement quantifié de grands modèles en FP4, tandis que l’article compare l’acquisition des faits rares selon la précision.

  3. Apparenté Base de cet article
    Deep Learning with Limited Numerical Precision
    Suyog Gupta, Ankur Agrawal, K. Gopalakrishnan et al. · 2015 · International Conference on Machine Learning · cité 2 339 fois · Pas encore résumé

    Deep Learning with Limited Numerical Precision étudie l’effet de la précision et de l’arrondi stochastique sur l’entraînement de réseaux, des principes de base mobilisés par l’article.

    Résumé scientifique

    Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithm…

  4. Apparenté Base de cet article
    Mixed Precision Training
    P. Micikevicius, Sharan Narang, Jonah Alben et al. · 2017 · International Conference on Learning Representations · cité 2 736 fois · Pas encore résumé

    Mixed Precision Training propose d’entraîner des réseaux en demi-précision avec des mécanismes compensatoires, fournissant un contexte antérieur à la comparaison BF16 et FP4.

    Résumé scientifique

    Deep neural networks have enabled progress in a wide variety of applications. Growing the size of the neural network typically results in improved accuracy. As model sizes grow, the memory and compute requirements for training these models also increases. We introduce a technique to train deep neural networks using half precision floating point numbers. In our technique, weights, activations and gradients are stored in IEEE half-precision format. Half-precision floating numbers have limited numerical range compared to single-precision numbers. We propose two techniques to handle this loss of information. Firstly, we recommend maintaining a single-precision copy of the weights that accumulate…

  5. Apparenté Même question
    How Do Large Language Models Acquire Factual Knowledge During Pretraining?
    Hoyeon Chang, Jinho Park, Seonghyeon Ye et al. · 2024 · Computation and Language · cité 134 fois · Résumé IA disponible

    How Do Large Language Models Acquire Factual Knowledge During Pretraining étudie l’acquisition et l’oubli des faits pendant le préentraînement, sans comparaison de précisions numériques.

    Résumé scientifique

    Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model's capability to acquire and maintain factual knowledge. Next, there is a power-law relationship between training steps and forgetting of memorization and ge…

Comment ces articles sont-ils trouvés ?

Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.

Mis à jour le 16 novembre 2026.

Le guide illustré des articles autour d’un article