Accuracy is not compatibility: two Evo 2 checkpoints hinge on one FP8 matrix multiplication

Xun Ding; Ziyuan Qin

AI summary · arXiv: 2610.39415v1 · cs.LG

Accuracy is not compatibility: two Evo 2 checkpoints hinge on one FP8 matrix multiplication

Xun Ding (Independent researcher), Ziyuan Qin (Emory University)

Original paper Submitted to arXiv on 29 Oct. 2026 18 pages, 3 figures, 1 table. Data: https://doi.org/10.5281/zenodo.23740390 ; code: https://github.com/astding/mmaemu License: NONEXCLUSIVE-DISTRIB 1.0 Also in: fr

How to read this summary? Illustrated guide

The essentials

Two Evo 2 checkpoints depend on reference FP8 accumulation rather than numerical accuracy; emulating their first projection largely restores model behaviour and BRCA1 variant scoring.

Question

Do fixed low-precision checkpoints preserve their function when accumulation arithmetic changes, including toward greater accuracy? The paper investigates whether Evo 2 failures on Blackwell GPUs can be localised and reversed without changing weights.

Method

The authors tested Evo 2 1B, 7B and 40B checkpoints on two Blackwell GPUs, using an H100 PCIe and RTX 4060 as reference platforms. Their mmaemu emulator reproduced all 168 captured H100 outputs from the 1B’s 21 FP8 projections, enabling selective arithmetic substitutions and controlled accumulation and residual perturbations. Evaluation used four upstream test prompts and 500 BRCA1 variants, including 111 loss-of-function variants, with 2,000 variant-bootstrap resamples.

Findings

Result Native FP8 execution failed the 1B and 40B forward tests on both tested Blackwell GPUs, whereas the 7B passed.
See the passage in the paper
Two of the three tested checkpoints fail on both sm_120 GPUs.
Result More accurate accumulation reproduced the failures even when the FP8 operands were unchanged.
See the passage in the paper
Both gave the native loss of 1.2480 in the 1B, and fp32 accumulation gave the native 1.4023 in the 40B: more accurate accumulation of the same operands, with no FP8 kernel involved, reproduces the failure (Fig. 1b).
Result For the 1B, emulating only the first projection restored the forward test, while leaving only that projection native caused failure.
See the passage in the paper
With only block 0’s projection computed by hopper-te and the other 20 native, the 1B passed its forward test (0.5027); with block 0 native and the other 20 emulated, it failed (1.2500).
Result In sampled 1B controls, small shrinkages of exact accumulations passed, but exact accumulation, larger shrinkages and enlargements failed.
See the passage in the paper
Only two of nine sampled gains on the exact accumulator passed, about 0.986 and 0.993; larger shrinkages of 2.1–4.2% (0.979, 0.972, 0.958), 1.000 and three enlargements failed (Fig. 2b).
Result Replacing only the first projection’s outputs raised 40B BRCA1 AUROC from 0.434 to 0.869 on the RTX 5070 Ti, despite narrowly missing the forward-test tolerance.
See the passage in the paper
For the 40B on the RTX 5070 Ti, the table raised AUROC from 0.434 to 0.869 (paired difference 0.435, 95% interval 0.360–0.507), with Pearson correlation 0.997 and 97.4% sign agreement against the H100 (Figs. 1c and 3e,f); full hopper-te emulation gave 0.998 and 97.4% (no 40B reference-pair band exists: the RTX 4060 ran out of memory). The table’s forward loss of 0.2170 still misses tolerance (Section 2.2): application agreement is distinct from forward-test compatibility.

Limitations

Generalisability is limited to one model family and the tested hardware, and the study does not establish whether training caused the dependence.
See the passage in the paper
The study concerns one model family, and whether training produced the dependence is not tested.
Biological evaluation covers one assay and one 500-variant cohort, with bootstrap intervals that exclude overlapping-window dependence and run-to-run variation.
See the passage in the paper
Biology is one assay and one 500-variant cohort; bootstrap intervals are conditional on the recorded runs and model neither overlapping-window dependence nor run-to-run variation.
Recovery of 40B variant scoring does not establish full forward-test or generation compatibility.
See the passage in the paper
The 40B table misses the forward tolerance on the RTX 5070 Ti, the PRO table’s forward loss was not recorded, and 40B generation was not tested under emulation or with the table.

Why it matters

The authors argue that numerical accuracy, operator agreement and application compatibility are distinct validation targets for fixed low-precision checkpoints. They recommend documenting reference arithmetic, execution provenance and application-level reference outputs rather than treating an FP8 format specification as sufficient for portability.

Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.

AI-generated keywords: FP8 arithmetic Evo 2 Hardware compatibility Accumulation precision Variant-effect prediction

Read further

Comprehensive Summary
Layman's Summary
Blog article
Original abstract

This paper examines whether low-precision neural-network checkpoints remain compatible with newer hardware that performs arithmetic more accurately. FP8 specifies how numbers are stored but not how products are accumulated during matrix multiplication. The authors study Evo 2, a DNA language model whose documentation specifies Hopper FP8 hardware for several checkpoints. Their central finding is that fixed model weights can depend on specific properties of reference arithmetic: more accurate accumulation can severely degrade predictions rather than improve them. The study compares Evo 2’s 1B and 40B checkpoints on an H100 reference GPU, an RTX 4060 where memory permits, and two Blackwell GPUs: the RTX 5070 Ti and RTX PRO 6000. The 7B checkpoint provides an additional comparison. The authors developed mmaemu, an emulator of reference FP8 matrix multiplication, and substituted its arithmetic into selected input projections while leaving the weights unchanged and all other computations native. The emulator reproduced all 168 captured H100 outputs from the tested 1B projections bit for bit. Evaluation combined the upstream four-prompt forward test, generation tests, and variant-effect classification using 500 BRCA1 variants, including 111 loss-of-function variants. Native Blackwell FP8 failed the forward tests for both 1B and 40B, whereas 7B passed. On the RTX 5070 Ti, cross-entropy losses were 1.2480 and 1.4023, respectively, compared with expected values near 0.5020 and 0.2159. The 40B model’s next-base accuracy fell to 25.7%, close to random guessing among four DNA bases, versus 91.7% on H100. Yet Blackwell’s arithmetic was substantially more accurate at the measured first projection: 99.996% of outputs matched correctly rounded values, compared with approximately 30% on the reference GPUs. Accumulating the same FP8 operands in standard FP32 also reproduced the failures, strengthening the attribution to accumulation arithmetic rather than operand quantization alone. Most sensitivity was localized to the first input projection. For 1B, emulating only this projection restored the forward-test loss to 0.5027 and generation identity to 70.95%, within the expected range. Leaving that projection native caused failure even when every later projection was emulated. Diagnostic interventions showed that multiplying exact accumulator outputs by approximately 0.985869—a 1.4% shrinkage matching the measured H100 gain—restored the 1B forward test. Exact sums, larger shrinkages, and increased accumulator precision failed. These gains were fitted in-sample and were not validated as general corrections. At a downstream residual-stream location, the structured difference introduced by native arithmetic caused performance to collapse, whereas five random perturbations of matched magnitude retained low losses. This indicates that perturbation magnitude alone was insufficient to explain the failure. Replacing the first projection with a lookup table of emulated reference outputs also recovered biological scoring. For 40B, BRCA1 classification AUROC increased from 0.434 to 0.869 on the RTX 5070 Ti, close to the H100 result of 0.868, and from 0.481 to 0.867 on the PRO 6000. However, the RTX 5070 Ti’s restored forward loss of 0.2170 narrowly missed the specified tolerance, demonstrating that application-level agreement and forward-test compatibility are distinct. The authors conclude that model releases should document reference accumulation arithmetic, software and hardware configurations, and application-level tolerances. They do not establish that training caused this dependence or that it is widespread. Limitations include the focus on one model family, limited reference hardware, no direct H100 captures of 40B projections, a single biological assay and variant cohort, and incomplete forward and generation testing of restored models. Bootstrap intervals describe uncertainty conditional on recorded runs, not broader biological or execution variability.

When More Accurate AI Hardware Makes Predictions Worse

Newer hardware usually promises a straightforward upgrade: faster computation, better efficiency, and sometimes more accurate arithmetic. But what happens when a neural network depends on the numerical quirks of older hardware? A study of the DNA language model Evo 2 shows that greater arithmetic accuracy can sometimes make predictions dramatically worse.

In “Accuracy is not compatibility: two Evo 2 checkpoints hinge on one FP8 matrix multiplication,” Xun Ding and Ziyuan Qin investigate this counterintuitive problem. Their finding is not that accurate arithmetic is undesirable. It is that fixed model weights may rely on specific computational behavior—and moving those weights to newer hardware can break that relationship.

The Hidden Compatibility Question in FP8

FP8 is a low-precision numerical format used to make neural-network computation more efficient. However, specifying FP8 does not fully specify how a calculation happens. It describes how numbers are stored, but not exactly how products are accumulated during matrix multiplication.

That distinction matters because a matrix multiplication combines many individual products. Different accumulation methods can produce different outputs, even when the stored weights and input values are identical. A checkpoint that works under one method may therefore behave differently under another.

The researchers examined Evo 2 checkpoints whose documentation specifies Hopper FP8 hardware. They compared the 1B and 40B models across an H100 reference GPU, an RTX 4060 where memory permitted, and two Blackwell GPUs: the RTX 5070 Ti and RTX PRO 6000. The 7B checkpoint supplied an additional comparison.

More Accurate Calculations, Worse Predictions

On native Blackwell FP8 arithmetic, both the 1B and 40B checkpoints failed the upstream four-prompt forward test. The 7B checkpoint passed, showing that the problem did not affect every tested checkpoint.

On the RTX 5070 Ti, the 1B and 40B cross-entropy losses reached 1.2480 and 1.4023, respectively, versus expected values near 0.5020 and 0.2159. Lower loss is better. For 40B, next-base prediction accuracy dropped from 91.7% on H100 to 25.7%—close to random guessing among the four DNA bases.

Yet Blackwell was substantially more accurate at the measured first projection: 99.996% of its outputs matched correctly rounded values, compared with approximately 30% on the reference GPUs. Accumulating the same FP8 operands in standard FP32 also reproduced the failures. This strengthened the case that accumulation arithmetic, rather than operand quantization alone, was responsible.

Tracing the Failure to One Projection

To isolate the cause, the authors developed mmaemu, an emulator of reference FP8 matrix multiplication. They substituted its arithmetic into selected input projections while keeping the weights unchanged and all other computations native. For the tested 1B projections, the emulator reproduced all 168 captured H100 outputs bit for bit.

Most of the sensitivity traced to the first input projection. Emulating only that projection restored the 1B forward-test loss to 0.5027 and generation identity to 70.95%, within the expected range. Leaving it native caused failure even when every later projection was emulated.

Why a Small Difference Was Not Necessarily Harmless

Diagnostic experiments offered another clue. Multiplying exact accumulator outputs by approximately 0.985869—a 1.4% shrinkage matching the measured H100 gain—restored the 1B forward test. Exact sums, larger shrinkages, and increased accumulator precision failed. However, these gains were fitted in-sample and were not validated as general-purpose corrections.

The researchers also tested whether the size of the numerical disturbance explained the collapse. At a downstream residual-stream location, the structured difference introduced by native arithmetic destroyed performance, while five random perturbations of matched magnitude retained low losses. The direction and structure of the change mattered, not simply its size.

Recovering Biological Scoring

The evaluation extended beyond forward tests to generation and variant-effect classification. The biological task used 500 BRCA1 variants, including 111 loss-of-function variants. Replacing the first projection with a lookup table of emulated reference outputs recovered much of the lost classification performance.

On the RTX 5070 Ti, 40B classification AUROC increased from 0.434 to 0.869, close to the H100 result of 0.868.
On the RTX PRO 6000, AUROC increased from 0.481 to 0.867.
The RTX 5070 Ti’s restored forward loss of 0.2170 still narrowly missed the specified tolerance.

That final result highlights an important distinction: recovering application-level performance does not necessarily mean passing a numerical compatibility test.

What Model Releases Should Document

The practical lesson is that a checkpoint’s compatibility requirements may extend beyond its weights and storage format. Releases should document reference accumulation arithmetic, software and hardware configurations, and application-level tolerances.

The evidence remains bounded. The study covers one model family, limited reference hardware, and one biological assay and variant cohort. It includes no direct H100 captures of 40B projections, and restored models received incomplete forward and generation testing. Bootstrap intervals describe uncertainty conditional on recorded runs, not broader biological or execution variability.

The authors do not establish that training caused this dependence or that it is widespread. They do demonstrate a consequential possibility: more accurate arithmetic is not automatically compatible arithmetic. For some checkpoints, preserving behavior requires understanding how the numbers were computed—not just how they were stored.

Created on 02 Nov. 2026

Was this synthesis useful?

Score: 0

Ask questions about this paper to our AI assistant

How to chat with this paper? Illustrated guide

Compare with the PDF

You can also chat with multiple papers at once here.

AI assistant instructions?

Tools for this paper

Explain it differently 3 free rewrites left today

Save to a collection Group papers in collections to compare them, chat with them and export them. Create a free account Log in

Export BibTeX · RIS (Zotero, Mendeley, EndNote) · Obsidian (Markdown) · Notion (Markdown)

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero

Citation alerts Citation alerts guide

Who works on it? The people and institutions who publish on its topic

Weekly letter Presented in "Artificial intelligence this week" of 2 November 2026. Artificial intelligence this week: when better scores hide the wrong answer

Journal club Premium+ Slides to present it Propose for the journal club

Versions What changed between versions

Talk about it in public Premium+ Press kit: release, questions, hype check Press kit guide

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.

Accuracy is not compatibility: two Evo 2 checkpoints hinge on one FP8 matrix multiplication

The essentials

Question

Method

Findings

Limitations

Why it matters

Read further

Definitions

When More Accurate AI Hardware Makes Predictions Worse

The Hidden Compatibility Question in FP8

More Accurate Calculations, Worse Predictions

Tracing the Failure to One Projection

Why a Small Difference Was Not Necessarily Harmless

Recovering Biological Scoring

What Model Releases Should Document

Ask questions about this paper to our AI assistant

Tools for this paper

Advanced tools Premium

Papers around this one

Closest papers

Prior work it builds on

Go further

Cite this paper (BibTeX)