Accuracy is not compatibility: two Evo 2 checkpoints hinge on one FP8 matrix multiplication
How to prepare an accurate paper press kit? Illustrated guide
Xun Ding, Ziyuan Qin · arXiv: 2610.39415v1 · Summary page
Why cover this story this week?
When more accurate arithmetic breaks a model
A model checkpoint contains weights, but those weights may not be enough to reproduce its behaviour. A study of Evo 2, a DNA language model, found that its 1B and 40B checkpoints failed the developers' forward test on Blackwell hardware, while the 7B passed. The surprising culprit was not insufficient numerical accuracy: more accurate accumulation also broke the affected models. Their behaviour depended on the arithmetic of a particular FP8 matrix multiplication on Hopper hardware.
The researchers built an emulator of that arithmetic and traced the dependence to an early projection. Emulating only that projection, without changing weights, raised the 40B model's BRCA1 variant-classification AUROC from 0.434 to 0.869 on an RTX 5070 Ti, close to the H100 result of 0.868. The study concerns a single model family and does not establish how the dependence arose during training. Still, it makes a concrete reproducibility argument: distributing weights may need to be accompanied by a specification of the arithmetic they expect.
Editorial context written by AI from abstracts, not independent scientific validation. The kit and its verified quotes rely on the source paper, not on this letter.
Talk about a paper in public without overstating it: yours, one you are asked to comment on, or one you write about as a journalist.
- Your paper: a draft release for your press office, the questions journalists will ask with what the paper lets you answer, the likely misreadings, a lay summary and a short thread.
- Asked to comment: what the paper shows and does not, its strengths and weaknesses with their passages, and what would settle its main claim.
- Writing about it as a journalist: reporting angles, the wording to check before publication, and the questions to ask the authors.
- For everyone: findings with their passages checked word for word, the novelty set against the closest earlier papers, a hype check, figures with their credit, and researchers for an outside view who have not signed with the authors.