Same Matmul, Different Bits
Two accelerators, one operation, and the function nobody wrote down
Take the operation a large language model runs more than any other: a bf16 matrix multiply with fp32 accumulation. Run it on an NVIDIA H100 and on an AWS Trainium 2, with the same inputs, on dot products 4,096 terms long. The two chips return identical bits in 0.4% of cases.
Neither chip is broken. Both are deterministic, both are reasonable, and both vendors describe what they compute in the same five words. The trouble is in what those words leave out.
The words hide a choice
Open an ISA manual and you will find sentences like “fadd adds two fp32
values.” That sentence hides two operations. One is the mathematical sum,
which is the same on every chip. The other is the choice of which fp32 value
stands in for that sum, and that is where chips are free to differ.
A single add makes one such choice, and IEEE 754 pins it down. A matrix unit makes thousands. It decides how to group the products, whether to line them up against the largest one, how many bits to keep, when to round, and in which direction. The standard says nothing about a unit that adds sixteen products at once, and the vendor documentation says “fp32 accumulation.” Everything that separates the two chips lives in the gap between those two sentences.
So which chip is right? The question has no answer, because there is nothing to compare against. I have written before about correctness without a reference, where the missing reference was a model’s output. It is stranger to find the same problem in arithmetic, the one place everybody assumed a reference existed.
Why it needs an answer
The practical question behind this series is simple to state. When a model trained on NVIDIA hardware runs on Trainium, does it lose quality because of the hardware’s numerics?
Answering it needs an experiment in which only the numerics change. That means running NVIDIA’s arithmetic, bit for bit, on Trainium. And that needs a description of NVIDIA’s arithmetic precise enough to execute. Prose is not precise enough. Neither is a kernel’s source code, because the arithmetic does not live in the source. It lives below it, in instructions whose behavior is not documented.
The thesis
A kernel’s output bits are a total, deterministic function of its input bits. Nobody had written that function down, so the project behind this series built a small language to do it. I will call what it produces a bit-exact spec.
The language rests on a few commitments. Values are exact, and a float appears only where a rounding is written out with its format, direction and rules for the awkward cases. The order of every reduction is fixed by the shapes of the tensors, never by the hardware it runs on. Every vendor instruction, including the tensor core, is a small macro over that core, and each macro records how much evidence stands behind it. The project has a field guide that teaches the language one idea per chapter, with figures that run both chips’ models in your browser.
What is in the series
- Fix the tree, free the schedule introduces the language: exact values, one lossy operator, and reduction order as a typed value that the launch configuration cannot touch.
- Two chips, written down uses it on the H100 tensor core and the Trainium 2 matrix unit, and finds that one of them leans toward zero.
- Check, then compile asks what the specs make possible: checking a kernel against its spec, and replaying one chip’s arithmetic on the other.
Where it ends up
None of this says Trainium is better than the H100, or worse. On long dot products Trainium’s matrix unit is in fact the more accurate of the two, by more than an order of magnitude. Accuracy is not the question. A model trained under one chip’s arithmetic has adapted to that arithmetic, and whether the other chip’s arithmetic costs it anything is something to measure, not argue. The language exists to make that measurement possible.
Let me be accurate on the title. “Same matmul” is the misleading half. The two chips never ran the same operation; they ran two operations with the same name. A spec is how you tell them apart.