Two Chips, Written Down
The H100 tensor core and the Trainium 2 matrix unit as specs, and the one that leans toward zero
A language for arithmetic is only vocabulary until it describes a real chip. This chapter writes down the matrix units of the H100 and the Trainium 2 in the language from the previous chapter, runs them side by side, and follows one finding to its consequence.
Two ways to say “fp32 accumulation”
The H100’s tensor core consumes a bf16 dot product in groups of sixteen products. Each product is exact. The group, together with the running accumulator as one more term, is lined up against its largest exponent. Every term is cut to a 25-bit window below that exponent, the kept parts are summed exactly, and the result is rounded once, toward zero, to fp32. Then the next group starts.
The Trainium 2 matrix unit does something else entirely. It splits the sum into four blocks of 32. Within each block it adds one term at a time, rounding to nearest even after every add, in an accumulator with fp32’s precision and a wider exponent. The four block sums combine pairwise and are rounded to fp32 once at the end.
Each description is a few lines in the language, and side by side the difference is one of kind. The H100 rounds rarely and toward zero, after cutting terms to a window. Trainium rounds at every add, to nearest, with no window. Both are described in their documentation as a bf16 matmul with fp32 accumulation.
How a black box becomes a macro
No vendor documents any of this, so each description had to be recovered from the hardware. For the H100, prior work had done much of it: MMA-Sim reverse engineered NVIDIA’s matrix units and proposed the window width, the group size and the rounding mode. The project did not take those parameters on faith.
It swept them. Window widths from 23 to 27 bits, group sizes of 4, 8, 16 and 32, rounding toward zero or to nearest even: forty candidate models, each run against the real H100 on chained dot products. Exactly one matched 100% of the outputs: a 25-bit window, groups of sixteen, round toward zero. The runner-up, a 26-bit window, matched 94% on ordinary inputs and 50% on inputs with spread exponents.
The runner-up matters as much as the winner. It shows the comparison can tell two nearby models apart, so a 100% match means something. That process, sweeping a parameter space and demanding a unique exact match, is the general method for turning a black box into a macro. Trainium had no prior model, so its macro came from new probing, and was then checked against more than two billion hardware outputs with zero mismatches.
Toward zero, every time
Here is the finding I want to follow: the H100 tensor core rounds toward zero. One rounding error is small, at most one unit in the last place. A dot product of 4,096 terms makes hundreds of them. With round-to-nearest, the errors point up and down and mostly cancel. With round-toward-zero, they all point the same way, and they add up.
The window makes it worse. Cutting a term to a window always shrinks its magnitude. In a long dot product the accumulator grows to many times a typical product, and from then on it is the largest term in every group, so every product is cut against it.
The measured result, at K = 4,096: the H100’s mean error is about 128 units in the last place, and most errors point toward zero. Trainium’s mean error is 4.6 units, with no preferred direction. The two chips agree on the exact bits 0.4% of the time.
This is where the obvious reading goes wrong. Trainium’s arithmetic is the more accurate of the two by more than an order of magnitude, but that is not the same as “Trainium runs NVIDIA-trained models better.” A model trained on the H100 was trained under the H100’s bias, and Trainium does not reproduce it. Whether that matters for the model’s quality is an empirical question, and nobody can answer it by staring at error statistics. It needs the experiment this whole effort is built for: replay the H100’s arithmetic on Trainium exactly, and attribute whatever gap remains to something else.
A macro is a hypothesis
A description recovered from hardware is a claim about hardware, so the language makes each one say how strong its evidence is. Every instruction macro carries a provenance record: documented, probed, or exhaustive. A billion matching outputs is a strong empirical claim, and it is still not a proof. An exhaustive check over all 2³² inputs of a one-argument fp32 function is a proof, for the configuration tested. A spec’s trust base is the list of probed records it reaches, and the checker can print it.
The exhaustive form is how the language handles functions nobody can write
down. Trainium’s exp is off by more than one unit in the last place on over
3% of inputs, by an algorithm no one has recovered, yet a spec can still name
it exactly, as a lookup table identified by its hash.
Kernels are built from these macros, and a kernel spec is one expression per output. Writing them down has a way of finding things. The first spec of Trainium’s RMSNorm matched the hardware on normal data and failed on extreme inputs. The cause was a single instruction: one of Trainium’s multiplies returns +0 for every zero result, where IEEE 754 says a negative number times zero is −0. Different bits. After a separate macro for that instruction, RMSNorm, softmax, SwiGLU and a GEMM all matched their specs with zero mismatches. A spec that is basically right would have hidden exactly these cases, and they are the cases this study is about.
Both vendors call it fp32 accumulation. Written down, it is two different functions, and one of them leans.
References
- Peichen Xie, Shuotao Xu, Yang Wang, Fan Yang, and Mao Yang. Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy. arXiv:2511.10909, 2025.
- Bit-Exact Field Guide, chapters 4, 7 and 9. 2026.