What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
How to prepare an accurate paper press kit? Illustrated guide
Chang Lu, Mengjiao Zhang · arXiv: 2611.12832v1 · Summary page
Why cover this story this week?
Cheap arithmetic can conceal expensive forgetting
Lower-precision arithmetic promises cheaper model training, but an almost unchanged loss curve may conceal a specific learning deficit. Researchers trained transformers on synthetic biographies with controlled exposure counts, both alone and mixed into web text. NVFP4 needed 1.32 times and 1.52 times as many exposures as BF16 to reach comparable factual recall, despite validation-loss differences of only 0.5% and 2%. The gap persisted when evaluation used BF16 arithmetic, pointing to a training effect rather than just a readout problem.
Controlled noise experiments implicated degraded gradient information: making updates noisier could impede rare-fact learning while barely disturbing the aggregate loss. Increasing the learning rate did not recover the cost. These experiments do not establish how much factual knowledge every large production model loses, but they identify a blind spot in efficiency comparisons. Matching average predictive performance need not mean learning infrequently encountered information equally well.
Editorial context written by AI from abstracts, not independent scientific validation. The kit and its verified quotes rely on the source paper, not on this letter.
Talk about a paper in public without overstating it: yours, one you are asked to comment on, or one you write about as a journalist.
- Your paper: a draft release for your press office, the questions journalists will ask with what the paper lets you answer, the likely misreadings, a lay summary and a short thread.
- Asked to comment: what the paper shows and does not, its strengths and weaknesses with their passages, and what would settle its main claim.
- Writing about it as a journalist: reporting angles, the wording to check before publication, and the questions to ask the authors.
- For everyone: findings with their passages checked word for word, the novelty set against the closest earlier papers, a hype check, figures with their credit, and researchers for an outside view who have not signed with the authors.