<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="/naizhengtan_blog/feed.xml" rel="self" type="application/atom+xml" /><link href="/naizhengtan_blog/" rel="alternate" type="text/html" /><updated>2026-11-09T00:45:01-05:00</updated><id>/naizhengtan_blog/feed.xml</id><title type="html">Cheng Tan</title><subtitle>Personal blog of Cheng Tan. Notes, mostly on systems, verification, and faithful LLM systems.</subtitle><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><entry xml:lang="en"><title type="html">Keep It Close</title><link href="/naizhengtan_blog/posts/keep-it-close/" rel="alternate" type="text/html" title="Keep It Close" /><published>2026-11-09T00:30:00-05:00</published><updated>2026-11-09T00:30:00-05:00</updated><id>/naizhengtan_blog/posts/keep-it-close</id><content type="html" xml:base="/naizhengtan_blog/posts/keep-it-close/"><![CDATA[<p>This is the third entry in a series on ideas that computer science keeps
rediscovering under new names, after
<a href="/naizhengtan_blog/posts/the-second-time-is-cheaper/">deduplication</a> and
<a href="/naizhengtan_blog/posts/everything-is-a-search-problem/">search</a>. This one
is older than both, and it sounds too simple to be a principle: <strong>keep the data
close to where it is used.</strong> Most of performance engineering, I’ve come to
believe, is this one idea applied at a new distance.</p>

<h2 id="performance-is-mostly-geography">Performance is mostly geography</h2>

<p>The reason is physics. A signal cannot cross a wire faster than light, and in one
cycle of a 3 GHz clock light travels about ten centimeters. Real on-chip wires
are much slower than that, because their delay grows with the square of their
length. So distance is time. It is also energy: on a 45 nm chip, a 32-bit
floating-point add costs about 0.9 picojoules, and reading the same 32 bits from
DRAM costs about 640. Arithmetic is cheap. Moving the operands to the arithmetic
is what you pay for.</p>

<p>Programs are kind to us here. They tend to reuse what they just touched and to
touch what sits next to it. Peter Denning named this <em>locality</em> in the late
1960s, when virtual-memory systems started <em>thrashing</em>, and his working-set
model explained why: a program needs only a small, slowly changing set of pages
at any moment, so keep that set close and the rest can live far away.</p>

<p>The same move then reappears at every scale. Caches keep recently used memory
next to the core. In the 1980s, Lisp machines built hardware for a language made
of pointers: CDR-coding stored a list’s cells next to each other, and their
garbage collectors copied related objects onto the same page to keep data local.
By the early 1990s they had lost to general-purpose workstations that ran Lisp
as fast or faster.</p>

<p>AI kernels are the newest instance. FlashAttention did not change the attention
computation. It changed where the intermediate result lives: instead of writing
the N×N attention matrix to GPU memory and reading it back, it computes in tiles
that stay in on-chip SRAM. GPT-2 trained three times faster. Kernel fusion in
general is the same trade: the same arithmetic, fewer trips.</p>

<p>None of these change <em>what</em> is computed. They change <em>where</em> the data is when it
is computed.</p>

<h2 id="every-copy-is-a-promise">Every copy is a promise</h2>

<p>How do you get data close? You copy it. A cache line is a copy of a memory block.
A replica is a copy of a database. A page in a buffer pool is a copy of a page on
disk. And the moment there are two copies, they can disagree.</p>

<p>So the second wave follows the first, always. Multiprocessors got private caches
and then needed coherence protocols; the MESI states date from 1984. Replicated
services needed consistency, and linearizability gave it a definition in 1990.
Databases that let transactions read their own snapshots needed isolation
levels, and a long argument about which anomalies are acceptable. Different
names, one problem: <strong>locality is copying, and consistency is the bill.</strong></p>

<p>The bill is not always due, and when it is not tells you something. FlashAttention
pays nothing for its tiles, because nobody else ever reads them. They are private
and short-lived. A read-only copy is also free: a thousand caches can hold the
same constant without a protocol. The bill arrives exactly when data is copied,
shared, and written. That gives a rule simple enough to check a design against:
copy what nobody writes; be careful about copying anything that someone else
writes.</p>

<p><img src="/naizhengtan_blog/assets/img/topic20/fig1.svg" alt="Every level of the memory hierarchy buys speed with a copy, and every shared copy needs a protocol to keep it agreeing with the others. A private copy, like a fused kernel's tile, needs none. Latencies are approximate." /></p>

<h2 id="moving-the-data-away-on-purpose">Moving the data away on purpose</h2>

<p>The cloud has spent the last decade doing the opposite: moving data <em>away</em> from
compute. Amazon Aurora separates the database engine from its storage; only redo
log records cross the network, and the storage keeps six copies across three
availability zones. CXL memory pools pull DRAM out of individual servers, because
up to a quarter of it sits stranded and memory can be half of a server’s cost.
The price is distance: Pond, Microsoft’s CXL pool design, measures 70 to 90
nanoseconds added to each access.</p>

<p>Why give up locality? For utilization, elasticity, and failure independence,
which locality never offered. And one far-away source of truth is, for a while,
simpler to keep consistent than many nearby copies.</p>

<p>The while does not last. Aurora’s compute nodes still keep a buffer cache, so
most reads never leave the machine. CXL 3.0 lets hosts share pooled memory
coherently, which means the pool can reach back and invalidate a host’s cached
lines. Disaggregate, and the latency pushes you to cache; cache, and the copies
send their bill. The pendulum swings between locality and a single truth, and
each swing reinvents the protocol of the last, at a larger distance.</p>

<p>This is where a piece of my own work lives. When the protocols that keep copies
agreeing become too complex to trust, you can check the outcome instead: record
what clients observed and ask whether some serial order explains it. That is
<a href="/naizhengtan_blog/posts/consistency-checking-is-the-new-verifier/">consistency checking</a>,
and it exists because locality keeps making copies.</p>

<h2 id="the-context-window-is-a-cache">The context window is a cache</h2>

<p>Now look at an AI agent. Its context window is the fastest memory it has, and
what it puts there is a copy: the file it read ten minutes ago, the test output
from the last run, the design it was told about this morning. Choosing what goes
in context is choosing a working set. Agents that retrieve well are agents with
good locality.</p>

<p>And their copies go stale. Another agent edits the file, a person changes the
plan, and the agent keeps acting on its own copy. With one agent, this is
annoying. With many agents sharing a repository, it is a coherence problem with
no coherence protocol. Re-reading a file before editing it is invalidation by
hand. I expect multi-agent systems to rediscover, in some form, the whole second
wave: ownership, invalidation, and an argument about which anomalies are
acceptable.</p>

<h2 id="close-and-agreed">Close, and agreed</h2>

<p>Keep it close is half of the rule. The other half is to know who else holds a
copy. Locality made computers fast, and consistency is what it costs to share
what you made fast. I don’t expect either half to go away. I expect the next
system, at whatever distance it works, to pay for both again.</p>

<h2 id="references">References</h2>

<ul>
  <li>Peter J. Denning.
<a href="https://doi.org/10.1145/363095.363141">The Working Set Model for Program Behavior</a>.
Communications of the ACM 11(5), 1968. See also
<a href="https://doi.org/10.1145/1070838.1070856">The Locality Principle</a>,
Communications of the ACM 48(7), 2005.</li>
  <li>Ron Ho, Kenneth W. Mai, and Mark A. Horowitz.
<a href="https://doi.org/10.1109/5.920580">The Future of Wires</a>.
Proceedings of the IEEE 89(4), 2001.</li>
  <li>Mark Horowitz.
<a href="https://doi.org/10.1109/ISSCC.2014.6757323">Computing’s Energy Problem (and What We Can Do about It)</a>.
ISSCC 2014.</li>
  <li>David A. Moon.
<a href="https://doi.org/10.1145/800055.802040">Garbage Collection in a Large Lisp System</a>.
ACM Symposium on LISP and Functional Programming, 1984.</li>
  <li>Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.
<a href="https://arxiv.org/abs/2205.14135">FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness</a>.
NeurIPS 2022.</li>
  <li>Mark S. Papamarcos and Janak H. Patel.
<a href="https://doi.org/10.1145/800015.808204">A Low-Overhead Coherence Solution for Multiprocessors with Private Cache Memories</a>.
ISCA 1984.</li>
  <li>Maurice P. Herlihy and Jeannette M. Wing.
<a href="https://doi.org/10.1145/78969.78972">Linearizability: A Correctness Condition for Concurrent Objects</a>.
ACM TOPLAS 12(3), 1990.</li>
  <li>Alexandre Verbitski et al.
<a href="https://doi.org/10.1145/3035918.3056101">Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases</a>.
SIGMOD 2017.</li>
  <li>Huaicheng Li et al.
<a href="https://doi.org/10.1145/3575693.3578835">Pond: CXL-Based Memory Pooling Systems for Cloud Platforms</a>.
ASPLOS 2023.</li>
  <li>CXL Consortium.
<a href="https://computeexpresslink.org/wp-content/uploads/2023/12/CXL_3.0_white-paper_FINAL.pdf">Compute Express Link 3.0 White Paper</a>.
2022.</li>
</ul>]]></content><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><category term="systems" /><category term="philosophy" /><summary type="html"><![CDATA[This is the third entry in a series on ideas that computer science keeps rediscovering under new names, after deduplication and search. This one is older than both, and it sounds too simple to be a principle: keep the data close to where it is used. Most of performance engineering, I’ve come to believe, is this one idea applied at a new distance.]]></summary></entry><entry xml:lang="en"><title type="html">Accountable by Construction</title><link href="/naizhengtan_blog/posts/accountable-by-construction/" rel="alternate" type="text/html" title="Accountable by Construction" /><published>2026-10-31T09:00:00-04:00</published><updated>2026-10-31T09:00:00-04:00</updated><id>/naizhengtan_blog/posts/accountable-by-construction</id><content type="html" xml:base="/naizhengtan_blog/posts/accountable-by-construction/"><![CDATA[<p>I give an agent a task. Later it says the task is done. How would I know?</p>

<p>Take a small example: write a correct function that sorts an array. In
version one, the agent replies “done, tested.” In version two, it writes the
code, writes tests, runs them, compares the outputs against what a sorted array
should look like, and reports that every check passed.</p>

<p>Almost everyone trusts version two more. But every step was done by the same
agent. The tests could be trivial, the run invented, the comparison simply
asserted. If the agent can err or lie in version one, it can in every step of
version two. So why does version two feel better? The intuition is right, and
the reason it is right tells us how to build accountability for AI instead of
hoping for it.</p>

<h2 id="not-more-trustworthy-more-checkable">Not more trustworthy, more checkable</h2>

<p>Humans make mistakes too, and we never fixed that by finding people who
don’t. We arranged work so that mistakes leave evidence. Version two does
three things version one does not. It <strong>splits one claim into smaller ones</strong>:
“the code is correct” is hard to check, “this test covers the empty array” is
easy. Some of those claims are executions, and a computation can be rerun by
someone else. And the steps leave artifacts, a test file and a run log, that
exist apart from the agent’s account of them.</p>

<p>None of this makes the agent more truthful. It changes what a lie costs. A
false “done” costs one sentence. A false version two needs a test file, a
log, and outputs that agree with each other and with the code, and survive
whatever check I choose to run afterward. Trust does not track how much work
the agent did. It tracks how much of that work I can check, and how cheaply.</p>

<h2 id="commit-first-then-check">Commit first, then check</h2>

<p><em>Whatever check I choose afterward</em> is where a Merkle hash tree earns its
place. Each node stores the hash of its children, so one short root value
commits to the whole structure; change anything and the root changes. Git
works this way, and so do the public logs that make web certificates
auditable.</p>

<p>Hash every artifact the agent produces, and every claim together with the
hashes it rests on. The agent hands over the root hash <em>before</em> I choose what
to open. Now a spot check means something: any step the agent faked had to be
faked in advance, and every one I open that does not match its commitment is
caught. Random sampling becomes an audit with computable odds.</p>

<p>A hash proves that what I see is what was committed, not that it is right.
Whether a claim is right depends on what kind of step the node records.</p>

<h2 id="three-kinds-of-node">Three kinds of node</h2>

<p><img src="/naizhengtan_blog/assets/img/topic19/fig1.svg" alt="An evidence tree for &quot;write a correct sorting function.&quot; Executed steps can be replayed and score 1.00; judged steps get a score from an independent certificate model; artifacts are committed by hash. Each node hashes its children, so the root hash commits to the whole tree before anyone chooses what to check. Scores are illustrative." /></p>

<p><strong>Executed</strong> nodes record a computation: run these tests on this code,
compare these outputs with this oracle. Anyone can replay them from the
committed inputs, so their credibility is in effect 100%. What they do not
cover is how the computation came to be: the script that runs the tests was
written by the agent, so “this script runs these tests” is a judged claim.
Running it is not.</p>

<p><strong>Judged</strong> nodes record claims no computation settles. Do these tests cover
what “sorted” means? Did the agent think of duplicates, the empty array,
reversed input? A separate <strong>certificate model</strong> reads the task, the artifacts,
and the claim, and assigns a credibility score. It must be independent of the
agent, with different weights and nothing but the committed artifacts to go
on: an agent testing its own code shares its own blind spots. And its scores
are judgments, not proofs; language models are still unreliable step-level
judges (see <a href="/naizhengtan_blog/posts/does-it-work/">Does It Work?</a>).</p>

<p><strong>Given</strong> nodes are the specification: “output is sorted and is a permutation
of the input.” Nobody checks it, because it defines the task. A person has to
read it and agree it is what they meant.</p>

<p>The root is the human’s task, children are subtasks, and each node carries a
kind and a score. Scores compose by product, so one weak link drags down
everything above it.</p>

<h2 id="what-the-human-reads">What the human reads</h2>

<p>Nobody reads the whole tree. A human reads the specification and the few
nodes with low scores. Everything else is replayed by machines, scored by an
independent model, or sampled against a commitment made in advance.</p>

<p>Machine-checked proofs already work this way: you trust a small checker and
the statement at the top, not the person who wrote the proof. And <a href="/naizhengtan_blog/posts/the-old-bargain/">every
verified system</a> still has a
paragraph at the top that someone read and believed. The difference is that an
evidence tree is made of steps a person can read, so a failure lands on a
named node.</p>

<h2 id="when-the-tree-is-too-big-to-read">When the tree is too big to read</h2>

<p>A sorting tree has a handful of nodes. The tree for porting a large system,
or for a week of agent work, has thousands, and “read the low-scoring nodes”
can still mean hundreds. Reading the tree has become a task. So treat it like
one: give it to an agent, and ask for an evidence tree.</p>

<p>The tempting request is “summarize this tree,” and it is the wrong one. A
summary that replaces evidence can lose the one bad node inside “mostly
fine.” Ask something narrower: does tree <em>T</em>, committed at root hash <em>h</em>,
support claim <em>R</em>, and where are its exceptions?</p>

<p>That question has a useful property. Most of it is computation over committed
data. Does every executed node replay? Does every hash check? Which judged
nodes score below 0.9? These are programs run over the tree, so in the new
tree they are executed nodes, replayable by anyone. What stays judged is
small: the exceptions themselves, and the threshold that defines an exception,
which belongs to the human. <strong>The tree about the tree is mostly executed, and
only the exceptions climb.</strong></p>

<p><img src="/naizhengtan_blog/assets/img/topic19/fig2.svg" alt="Reading a large evidence tree is itself a task. Level 0 is the work: thousands of nodes, a few judged ones weak. Level 1 asks whether the committed tree supports the claim: replaying, hash checks, and score filtering are computations over the tree, so they are executed nodes; only the exceptions are judged. Level 2 is what reaches the human: the specification and a few exceptions. Every level points at the one below by hash. Numbers are illustrative." /></p>

<p>Repeat it, and each level is smaller than the last, because aggregates are
cheap to check and, if the work was good, exceptions are rare. Stop when what
remains fits in a person’s attention: the specification and a few nodes no
machine could vouch for. Every level points at the one below by hash, so the
human can still open any leaf. Nesting shrinks what you must read, not what you
can check. Organizations already work this way: an auditor does not read every
invoice; she tests the controls, samples, and reports the exceptions.</p>

<p>This also has a limit, and the limit is informative. Nesting converges only
when judgment is sparse. If most of the original work is judgment, every level
forwards the same uncertain claims upward and nothing shrinks. That is not a
failure of the method. It is a measurement: work whose evidence cannot be
compressed is work we cannot yet delegate with accountability, and it marks
where people still have to do the thinking.</p>

<h2 id="leaving-evidence">Leaving evidence</h2>

<p>Proof-carrying code had the same economics decades ago: the producer builds
the evidence, which is expensive, and the consumer checks it, which is cheap.
AI makes the producer’s side nearly free, so we should ask for much more of it.</p>

<p>Version two was the right instinct, half-built. Don’t ask whether the agent is
trustworthy. Ask for a tree you can hold it to: every artifact committed,
every computation replayable, every judgment scored by someone else, and a top
small enough to read. Accountability for AI will not come from AI that never
errs. It will come from AI that cannot err without leaving evidence.</p>

<h2 id="references">References</h2>

<ul>
  <li>Ralph C. Merkle.
<a href="https://link.springer.com/chapter/10.1007/3-540-48184-2_32">A Digital Signature Based on a Conventional Encryption Function</a>.
CRYPTO 1987.</li>
  <li>George C. Necula.
<a href="https://dl.acm.org/doi/10.1145/263699.263712">Proof-Carrying Code</a>.
POPL 1997.</li>
  <li>Ben Laurie, Adam Langley, and Emilia Kasper.
<a href="https://www.rfc-editor.org/rfc/rfc6962">Certificate Transparency</a>.
RFC 6962, 2013.</li>
</ul>]]></content><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><category term="ai" /><category term="systems" /><category term="verification" /><summary type="html"><![CDATA[I give an agent a task. Later it says the task is done. How would I know?]]></summary></entry><entry xml:lang="en"><title type="html">C Is Back</title><link href="/naizhengtan_blog/posts/c-is-back/" rel="alternate" type="text/html" title="C Is Back" /><published>2026-10-19T01:00:00-04:00</published><updated>2026-10-19T01:00:00-04:00</updated><id>/naizhengtan_blog/posts/c-is-back</id><content type="html" xml:base="/naizhengtan_blog/posts/c-is-back/"><![CDATA[<p>For forty years, the history of programming languages has mostly been a
history of spending machine time to save human time. Garbage collection,
dynamic typing, rich standard libraries, virtual machines, layers of
frameworks: each one made programs easier to write and a little slower to
run. Python won the decade on exactly that bargain. C, which gives you the
hardware and very little else, kept the kernels and the databases and lost
most of the rest.</p>

<p>That bargain was right for its time. I think its time is ending. When an
agent writes the code, the human cost that high-level languages were built to
save mostly disappears, and the machine cost they charge does not. <strong>C is
back</strong>, or at least the case for it is.</p>

<h2 id="every-abstraction-was-a-trade">Every abstraction was a trade</h2>

<p>Languages did not grow complicated by accident. Each convenience removes a
burden from the programmer and moves it into the runtime. A garbage collector
means you never decide when memory dies; the runtime decides, later, and
pauses to do it. A generic container means you never write a hash table; you
pay for indirection on every lookup. A virtual machine means you never think
about the processor; a JIT thinks about it at run time, after a warm-up you
wait through.</p>

<p>Put differently, high-level languages postpone decisions. They let the
programmer leave memory layout, lifetimes, and types unstated, and they bind
those decisions at run time instead (see
<a href="/naizhengtan_blog/posts/every-decision-has-a-binding-time/">Every Decision Has a Binding Time</a>).
Postponing costs cycles, but writing every decision down costs people, and
people were the scarce resource. C is the opposite design: simple, close to
the hardware, every decision made explicitly by whoever writes the code.
Dangerous, and blazingly fast.</p>

<h2 id="writing-it-down-is-cheap-now">Writing it down is cheap now</h2>

<p>Coding agents change both terms of that trade. The boilerplate that made C
tedious, hash maps, growable arrays, manual memory management, costs an
agent seconds. It does not need abstractions to keep a large program in its
head. So the main reason to postpone decisions is gone; you can afford to
bind them early again.</p>

<p>The other cost of C was bugs, and here I want to be precise. Agents still
write bugs, including the memory bugs C is famous for. What changed is that an
agent also runs, without complaint and without being reminded, the tools that
catch them: AddressSanitizer, UndefinedBehaviorSanitizer, differential
fuzzers, brute-force oracles. Humans knew those tools existed and mostly
skipped them, because they were slow and boring to set up. The danger in C
was never only that bugs were easy to write. It was that checking was
expensive, and nobody did enough of it.</p>

<h2 id="an-experiment-boomslang-in-c">An experiment: Boomslang in C</h2>

<p>To test this, I asked a coding agent to port one of my research systems.
Boomslang checks whether a database honored the isolation level it promised.
It reads a trace of what clients sent and received and searches for an
ordering of transactions that explains it. The problem is NP-complete in
general, so the checker is really a small compiler: parse traces in eight
formats, build a graph of what is known and what is uncertain, prune it, and
hand the rest to a SAT solver with a graph-acyclicity theory. Its published
artifact is about 24,000 lines of Java, plus MonoSAT, a 98,000-line C++
solver, called through JNI.</p>

<p>The port took two prompts: “Implement Boomslang in C language,” and a loop
telling the agent to keep going until the C version ran the same tests as the
Java one. Twenty-two hours later, with no message from me in between, it did.
A few days of short follow-up prompts later, the numbers on the artifact’s
own experiments were these:</p>

<ul>
  <li><strong>60× faster</strong> (geometric mean over 78 runs where both finish) and 48× less
memory. On TPC-C with 50,000 transactions, Java takes 173 seconds and 62 GB;
C takes 1.2 seconds and 0.63 GB.</li>
  <li><strong>4,900 lines of C</strong> at the first complete commit, against 23,900 lines of
Java and the 98,000-line solver. No dependencies beyond libc and pthreads.</li>
  <li><strong>The same verdicts</strong> as Java on all but one run.</li>
</ul>

<p>The full account, with every prompt, is in the
<a href="/naizhengtan_blog/assets/topic18/boomslang-c.md">porting notes</a>.</p>

<h2 id="where-the-speed-came-from">Where the speed came from</h2>

<p>A port that changes the language and the algorithms at once makes it hard to
credit either, so the C version has a flag that builds exactly the graph Java
builds. With the same graph, C is about 30× faster. That is the language: no
JVM start-up, no garbage collector, no JNI boundary, data laid out the way
the algorithm reads it. The remaining 2× (4.5× on large runs) came from
algorithmic changes the agent tried once it had a fast base and a reference
to compare against, such as a reachability structure whose memory grows with transactions
times sessions rather than with transactions squared.</p>

<p>The solver is the more interesting number. MonoSAT is a general SMT solver
with many graph theories. Boomslang needs exactly one, acyclicity. The agent
wrote a solver for that one theory: 887 lines. General-purpose libraries
exist largely because writing the specialized version was too expensive for a
person. When it is not, you write the 887 lines that do only what you need.</p>

<h2 id="the-precondition-is-a-judge">The precondition is a judge</h2>

<p>None of this would mean much if the C code could not be trusted, and this is
the part I find most important. Every claim of correctness rested on a
comparison with something independent: the Java artifact on thousands of
random traces, a brute-force serializability oracle on 40,000 more, a literal
reading of Adya’s thesis for every isolation level he defines. The agent built
most of these judges itself, before it needed them.</p>

<p>The judges also found the agent’s own worst bug: a solver error present since the first
commit, which never produced a wrong verdict in any test and surfaced a day
later as a heap overflow under AddressSanitizer. That is exactly the bug C is
feared for, and it was caught exactly the way C bugs should be caught.</p>

<p>The agent never had to be right on the first try. It had to be checkable.</p>

<h2 id="what-c-still-costs">What C still costs</h2>

<p>Let me admit the parts I hold loosely. The same argument favors Rust, which
catches at compile time what sanitizers catch at run time; “C is back” is
partly a claim about low-level languages in general, and C is simply
a representative case.
The C solver is still slower than MonoSAT on one deliberately
hard instance, because its branching heuristic is simpler.</p>

<p>So the remaining cost of C is review, and review is easier for 5,000 lines
than for 24,000 lines plus a dependency.</p>

<p>Let me be accurate: what is back is not nostalgia for C.
It is the old rule that the person who knows what the program needs should
write it down, now that writing it down no longer costs a person.</p>

<h2 id="references">References</h2>

<ul>
  <li>Jian Zhang, Shuai Mu, and Cheng Tan.
<a href="https://arxiv.org/abs/2604.20587">Making Transaction Isolation Checking Practical</a>.
ACM ATC 2026.</li>
  <li>Atul Adya.
Weak Consistency: A Generalized Theory and Optimistic Implementations for Distributed Transactions.
PhD thesis, MIT, 1999.</li>
  <li>Sam Bayless, Noah Bayless, Holger H. Hoos, and Alan J. Hu.
SAT Modulo Monotonic Theories.
AAAI 2015.</li>
</ul>]]></content><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><category term="systems" /><category term="ai" /><category term="programming" /><summary type="html"><![CDATA[For forty years, the history of programming languages has mostly been a history of spending machine time to save human time. Garbage collection, dynamic typing, rich standard libraries, virtual machines, layers of frameworks: each one made programs easier to write and a little slower to run. Python won the decade on exactly that bargain. C, which gives you the hardware and very little else, kept the kernels and the databases and lost most of the rest.]]></summary></entry><entry xml:lang="en"><title type="html">Loaded Dice</title><link href="/naizhengtan_blog/posts/loaded-dice/" rel="alternate" type="text/html" title="Loaded Dice" /><published>2026-10-04T09:00:00-04:00</published><updated>2026-10-04T09:00:00-04:00</updated><id>/naizhengtan_blog/posts/loaded-dice</id><content type="html" xml:base="/naizhengtan_blog/posts/loaded-dice/"><![CDATA[<p>In 1980 the Pennsylvania Lottery drew its Daily Number live on television:
physical ping-pong balls, rising from machines, in front of the camera. The
number was 6-6-6. Beforehand, the host and a few insiders had injected paint
into every ball except the 4s and the 6s, so only those could rise, and their
friends had bought every combination of 4s and 6s.</p>

<p>What strikes me is that <strong>nothing illegitimate came out of the machine.</strong> 666
is a valid lottery number, drawn from real balls. The attack forged no
outcome; it chose one.</p>

<h2 id="randomness-used-to-be-a-quiet-input">Randomness used to be a quiet input</h2>

<p>Attacks on randomness are not new. Most of the digital ones target
cryptography: a predictable generator or a reused random value, and a secret
key falls out.</p>

<p>But in ordinary software, randomness rarely decided what the program did. A
hash seed, a backoff jitter, a load balancer’s coin flip: change any of them
and the computation computes the same thing, slightly faster or slower.
Randomness was an input to secrecy and to performance, not to behavior.</p>

<h2 id="ai-runs-on-dice">AI runs on dice</h2>

<p>That is no longer true. An AI system draws random numbers at three layers
(see <a href="/naizhengtan_blog/posts/replay-comes-back/">Replay Comes Back</a>). At the
<em>tensor</em> layer, a reduction is summed in an order that depends on how many
requests share the batch. At the <em>token</em> layer, the sampler draws the next
token from a distribution. At the <em>trace</em> layer, the agent reads a clock, a
process id, a tool’s output, whatever the world happens to say that second.</p>

<p>None of these can simply be removed. Always taking the most likely token
produces text the paper that named the problem called “bland and strangely
repetitive.” Repeated sampling is how models get better answers: on one
coding benchmark, the share of problems solved by at least one sample rises
from 15.9% with one sample to 56% with 250. And determinism has a price. Making inference bit-for-bit
reproducible required batch-invariant kernels that, in the published
measurement, took a run from 26 seconds to 42 at best. OpenAI
documents its seed parameter with a sentence worth framing: “Determinism is
not guaranteed.”</p>

<p>So randomness is now load-bearing. It decides which words appear, which
command runs, which file gets deleted.</p>

<h2 id="a-new-attack-surface">A new attack surface</h2>

<p>Once randomness decides behavior, steering it is an attack, and the
literature already has examples at every layer.</p>

<p>At the token layer, simply changing decoding settings, with no adversarial
prompt at all, raised misalignment on eleven open models from 0% to over 95%.
Resampling randomly perturbed prompts until one gets through, “best-of-N
jailbreaking,” succeeded on 89% of attempts against GPT-4o with ten thousand
tries. Work from my group searches the decoding tree directly and finds
prompts that look safe under default settings but reach harmful outputs along
paths the sampler can legitimately take. And the paint-in-the-balls attack
has a literal analogue: a recent preprint shows that a compromised sampler’s
random number generator can inject chosen tokens while every logit stays
untouched.</p>

<p>At the tensor layer, the batch is a shared random variable. Two preprints
show that an attacker who lands in the same batch as a victim can recover the
victim’s prompt, through expert routing in one case and through batch-wide
quantization in the other. Both leak rather than steer, so far; steering is
the obvious next paper. At the trace layer, agent runs can fork on nothing but a timestamp. And nondeterminism is also good cover: an audit
that expects outputs to vary has a hard time telling variation from a
provider quietly serving a cheaper model.</p>

<h2 id="no-ground-truth-and-no-single-place-to-look">No ground truth, and no single place to look</h2>

<p>Two properties make these attacks different from the ones we know how to
handle.</p>

<p>First, <strong>there is no wrong answer to catch.</strong> A conventional attack produces
something checkably bad: a forged signature, a corrupted record, a value no
correct execution could produce. An attack on randomness produces an output
that some legitimate execution could have produced, so it passes any check
of the form “does a legitimate run explain this output?”
(see <a href="/naizhengtan_blog/posts/ai-correctness-without-a-reference/">Correctness Without a Reference</a>).
The attacker lives inside that existential quantifier. Every candidate is
legitimate; the attack is in which one was chosen. 666 passes every check a
lottery number can pass.</p>

<p>Second, the choice is hidden by depth. A tie broken differently in a GPU
reduction becomes a different token, which becomes a different shell
command, which becomes a different state of the machine. By the time the
damage is visible at the top, its cause is three layers down, and every layer
in between behaved legitimately given its input.</p>

<h2 id="trust-but-replay">Trust, but replay</h2>

<p>Prevention will not get us far. We cannot remove the randomness, and we
cannot forbid any outcome, because every outcome is allowed. What we can do
is what lotteries learned to do after 1980. In Florida today, for example,
the balls are weighed before and after each drawing, an independent accountant witnesses it, and a
difference of more than a gram sends the set to investigators. Nobody claims
the drawing cannot be rigged. The claim is that a rigged drawing leaves
evidence.</p>

<p>For AI the equivalent is a discipline in three steps. <strong>Log the draws</strong>: every
random value at every layer, the batch composition, the sampled tokens and
seeds, the clock and tool results. Then <strong>replay</strong>: given the log, reproduce
the run, and check that the randomness was what it claims to be, the same
way a randomness beacon publishes signed, hash-chained values anyone can
verify after the fact. Then <strong>attribute</strong>: when something went wrong, replay
it with one recorded draw replaced by another legitimate candidate. If the
harm disappears, that draw is the root cause. The same record-and-replay
machinery serves debugging (see <a href="/naizhengtan_blog/posts/replay-comes-back/">Replay Comes Back</a>);
here it faces an adversary instead of a bug.</p>

<p>The definition of root cause matters here, because it fits the attack
exactly. You cannot say the chosen draw was invalid; it was valid. You can
say that a different valid draw would not have done the harm, and that this
draw was the one that mattered. That is a counterfactual, and counterfactuals
need a recording.</p>

<p>Trust but verify, then, with one adjustment: for randomness, verification
does not mean checking the outcome. It means being able to replay the
choice.</p>

<h2 id="references">References</h2>

<ul>
  <li>Pennsylvania Lottery scandal, 1980: <a href="https://archive.triblive.com/news/666-an-infamous-mark-of-state-lottery-fix-in-1980/">TribLIVE</a>, <a href="https://www.wesanews.org/archives/2015-06-19/triple-six-fix-how-rigging-the-pa-lottery-inadvertently-contributed-to-its-success">WESA</a>, <a href="https://www.butlereagle.com/20241008/lottery-scandal-brought-down-beloved-tv-personality/">Butler Eagle</a>.</li>
  <li>Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.
<a href="https://arxiv.org/abs/1904.09751">The Curious Case of Neural Text Degeneration</a>.
ICLR 2020.</li>
  <li>Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini.
<a href="https://arxiv.org/abs/2407.21787">Large Language Monkeys: Scaling Inference Compute with Repeated Sampling</a>.
arXiv:2407.21787, 2024.</li>
  <li>Horace He and Thinking Machines Lab.
<a href="https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/">Defeating Nondeterminism in LLM Inference</a>.
Thinking Machines Lab: Connectionism, September 2025.</li>
  <li>Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen.
<a href="https://arxiv.org/abs/2310.06987">Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation</a>.
ICLR 2024.</li>
  <li>John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma.
<a href="https://arxiv.org/abs/2412.03556">Best-of-N Jailbreaking</a>.
NeurIPS 2025.</li>
  <li>Shuyi Lin, Anshuman Suri, Alina Oprea, and Cheng Tan.
<a href="https://naizhengtan.github.io/doc/papers/jo26shuyi.pdf">Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problem</a>.
MLSys 2026.</li>
  <li>Ziyang You, Xiaoke Yang, Zhanling Fan, Feng Guo, Xiaogen Zhou, and Xuxing Lu.
<a href="https://arxiv.org/abs/2605.08313">Seed Hijacking of LLM Sampling and Quantum Random Number Defense</a>.
arXiv:2605.08313, 2026.</li>
  <li>Itay Yona, Ilia Shumailov, Jamie Hayes, and Nicholas Carlini.
<a href="https://arxiv.org/abs/2410.22884">Stealing User Prompts from Mixture of Experts</a>.
arXiv:2410.22884, 2024.</li>
  <li>Hanna Foerster, Ilia Shumailov, Cheng Zhang, Yiren Zhao, Jamie Hayes, and Robert Mullins.
<a href="https://arxiv.org/abs/2604.26505">Dynamic Quantization Can Leak Your Data Across the Batch</a>.
arXiv:2604.26505, 2026.</li>
  <li>Will Cai, Tianneng Shi, Xuandong Zhao, and Dawn Song.
<a href="https://arxiv.org/abs/2504.04715">Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs</a>.
arXiv:2504.04715, 2025.</li>
  <li>OpenAI Cookbook.
<a href="https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter">Reproducible outputs with the seed parameter</a>
(page on the <code class="language-plaintext highlighter-rouge">seed</code> parameter).</li>
  <li>Florida Lottery.
<a href="http://flrules.elaws.us/fac/53er16-40">Rule 53ER16-40</a>.
Florida Administrative Code, 2016.</li>
</ul>]]></content><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><category term="systems" /><category term="ai" /><category term="verification" /><summary type="html"><![CDATA[In 1980 the Pennsylvania Lottery drew its Daily Number live on television: physical ping-pong balls, rising from machines, in front of the camera. The number was 6-6-6. Beforehand, the host and a few insiders had injected paint into every ball except the 4s and the 6s, so only those could rise, and their friends had bought every combination of 4s and 6s.]]></summary></entry><entry xml:lang="en"><title type="html">One Source, Many Papers</title><link href="/naizhengtan_blog/posts/one-source-many-papers/" rel="alternate" type="text/html" title="One Source, Many Papers" /><published>2026-10-01T09:00:00-04:00</published><updated>2026-10-01T09:00:00-04:00</updated><id>/naizhengtan_blog/posts/one-source-many-papers</id><content type="html" xml:base="/naizhengtan_blog/posts/one-source-many-papers/"><![CDATA[<p>Every paper I have written was compiled exactly once. I chose the section
order, the running example, the notation, and the figure that carries the
idea, all for one imagined reader, and the camera-ready deadline froze them.
Everyone afterward reads that same build: the ML researcher who has never
heard of an isolation level, the engineer deciding whether her team should
care, the student trying to reproduce it. Most are not the reader I imagined.</p>

<p>That used to be unavoidable: a second version cost as much as the first, and
nobody wrote twelve. It no longer is. So what we publish should change:
<strong>publish the source, and let each reader’s AI compile the paper for that
reader.</strong></p>

<h2 id="the-paper-is-the-source-the-reader-is-the-target">The paper is the source; the reader is the target</h2>

<p>The published artifact becomes a Markdown file that a reader hands to their
own assistant (Claude, ChatGPT, whatever they use) with one sentence:
<em>compile this paper for me.</em> The assistant asks how the reader takes in
technical material, which programming language they read, what field they
come from, and what they want from the paper, then builds one page for them.
If it already knows the reader well, it skips the questions and just
compiles.</p>

<p>The file has four parts: instructions on how to interview the reader, plan
the version, and emit it; a <strong>claim ledger</strong> listing every number and stated
limitation, which every version must keep exactly; the paper itself, the
authoritative text; and a fourth I will come back to.</p>

<p>The rules matter more than the prompts. Numbers are fixed. Limitations are
never dropped, however short the version. Anything the assistant adds (an
analogy, a worked example, a redrawn diagram) is marked as added, never
passed off as the authors’ words. Pseudocode may be translated into the
reader’s language, line for line and labeled. Plots are never redrawn with
invented data. The reader gets a different presentation, not different
claims.</p>

<h2 id="two-decisions-two-binding-times">Two decisions, two binding times</h2>

<p>An <a href="/naizhengtan_blog/posts/every-decision-has-a-binding-time/">earlier post</a>
sorted a system’s decisions by one question: <em>when does this stop being
open?</em> Ask it of a paper.</p>

<p>A paper makes two kinds of decisions. One is what it claims: the problem,
the mechanism, the numbers, the limits. The other is how it presents them:
order, example, notation, depth, figures. Today both bind at the same moment,
the camera-ready deadline, in the same compiler, the authors, which runs
once. Nobody chose to bind them together; there was simply no second
compiler.</p>

<p>The two want opposite binding times. Claims should bind early. Others build
on them, cite them, and try to break them, which only works if they hold
still, and authors can only stake their names on something fixed.
Presentation should bind late, because what it depends on (who is reading,
what they know, why they opened the paper) does not exist at write time. The
author can only guess. The reader’s assistant knows.</p>

<p>Programmers will recognize the move. An ahead-of-time compiler must emit one
binary for every machine; a just-in-time compiler waits for the actual
machine and workload, and specializes. Nobody thinks the JIT changes what the
program means. The same post named the seam that makes this safe: the
<em>interface</em>. One level defines it; the level below implements it and can
change without renegotiating anything above. In a paper, the ledger is the
interface and each rendering an implementation. Fix the interface early,
bind the implementation late, and check every implementation against the
interface.</p>

<p>So the proposal is not “let AI rewrite papers.” It is narrower: <strong>split the
paper’s two binding times.</strong> An assistant that already knows its reader is
the limit of this idea, the latest binding possible, with no interview at
all.</p>

<h2 id="one-paper-five-readers">One paper, five readers</h2>

<p>I tried it on my own paper. Cobra (OSDI’20) checks whether a black-box
database actually delivered serializability, and it is dense with graphs,
constraint encodings, and a pseudocode figure carrying the core algorithm. I
converted the published text into a source file, wrote the ledger (twelve
claims, eight limitations), and gave it to five fresh assistants, each seeing
only the file and a short reader profile. The
<a href="/naizhengtan_blog/assets/topic16/cobra.paper.md">source file</a> is
public; hand it to your own assistant and get a sixth.</p>

<p>The five papers differed in the ways I hoped:</p>

<ul>
  <li>An <a href="/naizhengtan_blog/assets/topic16/versions/ml-visual-python.html">ML postdoc</a>
who prefers diagrams and reads Python got eight thousand words and eleven
diagrams, with the hidden version order framed as a latent variable and GPU
pruning shown as what it is underneath, repeated matrix multiplication.</li>
  <li>A <a href="/naizhengtan_blog/assets/topic16/versions/systems-verbal-c.html">systems PhD student</a>
who prefers prose, reads C, and wants to judge the claims got the algorithm
in C, kernel analogies (lock-ordering checkers, RCU), and reading-group
questions on the evaluation.</li>
  <li>A <a href="/naizhengtan_blog/assets/topic16/versions/practitioner-visual-java.html">backend engineer</a>
with ten minutes and a Java background got less than half the length, six
diagrams, and a section titled “Should your team care?”</li>
  <li>A <a href="/naizhengtan_blog/assets/topic16/versions/pl-verbal-python.html">programming-languages researcher</a>
got the database vocabulary mapped onto memory-model relations and a list
of what it would take to rebuild the system.</li>
  <li>A <a href="/naizhengtan_blog/assets/topic16/versions/highschool-visual-python.html">high-school student</a>
with one Python class and almost no systems background. This reader is not
the paper’s audience, and the version drops most of the technical content.
But it is readable: a key-value store becomes a Python dictionary, a
transaction becomes moving ten dollars from Alice to Bob, and the problem
becomes a question a teenager can ask: can you trust a database you cannot
see inside? The alternative was never a simpler paper. It was an unreadable
one.</li>
</ul>

<p>What did not change is the point. Every number in all five versions traces to
the source. All five say the 10× improvement is verification cost, not
throughput, the sentence I would most expect a summarizer to get wrong. All
five keep the limitations, including the plainest: we did not find a
serializability violation in the wild.</p>

<h2 id="a-paper-that-keeps-reading-after-it-is-published">A paper that keeps reading after it is published</h2>

<p>The fourth part of the file is the one I most wanted.</p>

<p>A published paper is frozen at the literature it read; a reader opening Cobra
today cannot learn from it what happened next. So the source carries a
dated section labeled <em>after publication: not part of the paper</em>, not written
by the authors. For Cobra it records 72 citations on Semantic Scholar as of
September 2026 and the work that builds on it directly: snapshot-isolation
checkers reusing its encoding, weak-isolation checkers comparing against it,
a version adding SQL predicates with a reported 60–100× speedup, and others,
plus one paragraph on where the field stands. An assistant that can browse
may update it, as long as every new item carries a link and a date and stays
visibly separate from the snapshot.</p>

<p>Research already <a href="/naizhengtan_blog/posts/research-under-contention/">runs on stale reads</a>.
This lets a paper refresh its own.</p>

<h2 id="what-the-author-still-owns">What the author still owns</h2>

<p>The obvious objection: if every reader sees a different paper, what did the
authors publish, and what are they accountable for?</p>

<p>The source, and they stake their names on the ledger: the same answer I gave
about <a href="/naizhengtan_blog/posts/skin-in-the-game/">what stays valuable</a>
when generation gets cheap, the claim you would pay for if it were wrong.
Prose, order, examples, and diagrams are presentation, now cheap to
regenerate. Claims are not. Citation and review follow: cite the source and
its version, as you would a commit; review the source and its ledger. A
rendering is correct when the source could have produced it.</p>

<p>I will admit the cost. A version I never read can still go wrong in ways the
rules do not anticipate, such as an analogy that imports an assumption the
paper never made. The ledger catches altered numbers and dropped caveats, not
misleading metaphors. My bet is that this risk is smaller than the one we
already take: a single version most of its readers do not fully understand.</p>

<h2 id="an-intermediate-stage">An intermediate stage</h2>

<p>This is not the end state. A compiled paper is still a document you read from
the top. The version I want answers back: change a parameter and watch the
figure change, ask why a step holds and get the proof. That is a later post.</p>

<p>To conclude, I argue for a publication pattern that binds what a paper
claims once, early, by the people who stand behind it, and binds how it reads
late, once per reader. One source, many papers: not many truths, but one set
of claims and as many ways into them as there are readers.</p>]]></content><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><category term="research" /><category term="ai" /><category term="systems" /><summary type="html"><![CDATA[Every paper I have written was compiled exactly once. I chose the section order, the running example, the notation, and the figure that carries the idea, all for one imagined reader, and the camera-ready deadline froze them. Everyone afterward reads that same build: the ML researcher who has never heard of an isolation level, the engineer deciding whether her team should care, the student trying to reproduce it. Most are not the reader I imagined.]]></summary></entry><entry xml:lang="en"><title type="html">Token-Based Formal Methods</title><link href="/naizhengtan_blog/posts/token-based-formal-methods/" rel="alternate" type="text/html" title="Token-Based Formal Methods" /><published>2026-09-20T15:00:00-04:00</published><updated>2026-09-20T15:00:00-04:00</updated><id>/naizhengtan_blog/posts/token-based-formal-methods</id><content type="html" xml:base="/naizhengtan_blog/posts/token-based-formal-methods/"><![CDATA[<p>Engineers make correctness arguments every day: <em>this retry is safe because the
handler is idempotent; this queue cannot grow without bound because the
producer blocks on the same lock; the window never contains a repeated
character because we move the left edge past each repeat.</em> They show up in
design docs, code review, and comments above tricky loops. They are the
reasoning that actually keeps software working, and they are almost never
written in boolean logic.</p>

<p>That is awkward, because formal methods have spent fifty years getting very
good at checking correctness claims. They have proved kernels, compilers, and
distributed protocols, and the proofs hold. But they check arguments written
in a formal language, and the arguments above are not.</p>

<p>This series asks whether the gap can be closed from the other direction.
Instead of translating people into a solver’s language, can we make ordinary
engineering arguments checkable as they are?</p>

<h2 id="the-thesis">The thesis</h2>

<p><strong>A natural-language argument can be checked like a program proof.</strong> Break it
into small steps: two or three premises, one conclusion, one familiar
inference. Name what each step uses and what it produces. The result is a
proof graph. Its nodes can be checked one at a time, its wiring can be checked
by ordinary code, and when something fails, the failure points at a particular
claim rather than at the document as a whole.</p>

<p>The move underneath is narrower than it sounds, and I think it is the right
way to read the current wave of work. Hoare logic never proved the
implications inside a program proof; its rule of consequence hands them to
whatever language the assertions are written in. For fifty years that delegate
has been a decision procedure—a solver, a tactic library, a decade of
automation. Replace it with a reader, and you inherit both the promise and the
risk of this whole approach.</p>

<h2 id="why-now">Why now</h2>

<p>Because code production changed.</p>

<p>An agent returns a working diff in minutes. What it does not return is an
account of the alternatives it weighed, the invariant it was holding, or why
the boundary case is safe. Sometimes that reasoning was never written down;
often it never existed in a form anyone could inspect. You inherit the code
without its argument.</p>

<p>Tests do not fill the gap, because they never did. A suite reports on the
cases someone thought to write. That was always partial evidence, and it was
enough anyway—not because the sample was good, but because it was never the
whole case. There was also an author who could explain the design and a
reviewer who could weigh the explanation. The green check corroborated an
argument that lived somewhere else. As code gets cheaper and its author less
available, the passing test is asked to carry more trust than it can.</p>

<h2 id="what-is-in-the-series">What is in the series</h2>

<ul>
  <li><strong><a href="/naizhengtan_blog/posts/the-old-bargain/">The old bargain</a></strong> looks
at what formal methods deliver and what they charge: twenty person-years of
proof against two of kernel, a specification nothing checks, and a
certificate few people can read.</li>
  <li><strong><a href="/naizhengtan_blog/posts/token-based-logic/">Token-based logic</a></strong>
keeps Hoare’s structure and changes the assertion language. It argues for
small steps, and shows the one composition move that turns up as
Aristotle’s middle term, as Gentzen’s cut, and as sequencing in Hoare logic.</li>
  <li><strong><a href="/naizhengtan_blog/posts/does-it-work/">Does it work?</a></strong> tries the
idea on one algorithm and one planted bug, looks at the two systems
attempting this at scale, and reviews what is known about the reliability of
the judge.</li>
</ul>

<h2 id="where-it-ends-up">Where it ends up</h2>

<p>This does not replace Rocq, Lean, or SMT solvers, and the people building it
do not claim that it does.</p>

<p>The experiment in the last chapter is deliberately small: fifteen lines of
code, proved twice—once as sixteen natural-language steps, once as 445 lines
of machine-checked Rocq. The graph turned out to be a usable outline for the
formal proof. It predicted where the hard part would be. And when the code was
sabotaged, the failure landed on the node whose own justification had already
identified the crucial guard.</p>

<p>The result it can defend is the negative one: <em>this step breaks, and this
input proves it.</em> A counterexample is a fact, because the program ran. A
positive verdict is a stack of fallible judgments, and a stack of judgments is
only as good as its worst member. That limit is real, and I would rather state
it plainly than argue around it. What is left is still worth having—a
readable argument that fails in a named place, about a claim stated in the
language its author actually thinks in.</p>]]></content><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><category term="systems" /><category term="research" /><category term="ai" /><summary type="html"><![CDATA[Engineers make correctness arguments every day: this retry is safe because the handler is idempotent; this queue cannot grow without bound because the producer blocks on the same lock; the window never contains a repeated character because we move the left edge past each repeat. They show up in design docs, code review, and comments above tricky loops. They are the reasoning that actually keeps software working, and they are almost never written in boolean logic.]]></summary></entry><entry xml:lang="en"><title type="html">The Old Bargain</title><link href="/naizhengtan_blog/posts/the-old-bargain/" rel="alternate" type="text/html" title="The Old Bargain" /><published>2026-09-20T14:00:00-04:00</published><updated>2026-09-20T14:00:00-04:00</updated><id>/naizhengtan_blog/posts/the-old-bargain</id><content type="html" xml:base="/naizhengtan_blog/posts/the-old-bargain/"><![CDATA[<p>In 2009, NICTA finished a proof of the seL4 operating-system kernel. The
kernel is 8,700 lines of C and 600 lines of assembler; its Isabelle proof is
200,000 lines. Building the kernel took about 2.2 person-years. Proving it
took about twenty. Two years earlier, CompCert had done the same for a C
compiler: 42,000 lines of Coq, now the Rocq Prover, of which 76% is the proof
that compilation preserves what your program means. IronFleet later verified
two distributed systems and reported a detail that deserves to be better
known: both worked the first time they were run.</p>

<p>That is not a story of failure. It is a story of complete success inside a
boundary that has never moved. Almost all software is written outside it. Cost
explains part of that, and the rest is easier to see by looking at what Hoare
logic actually does.</p>

<h2 id="what-the-rules-do-and-what-they-quietly-dont">What the rules do, and what they quietly don’t</h2>

<p>Hoare logic uses a triple: a precondition, a program, a postcondition. Its
rules describe how programs compose: sequencing, assignment, iteration.
They are small and they are elegant. Then there is the rule of consequence: if
a program establishes R, and R implies S, then it establishes S.</p>

<p>Look at the second premise. That implication is not a fact about the program.
It is a fact about the assertion language, and Hoare logic does not prove it.
It assumes someone else can. The program rules supply structure; the
mathematical content sits in the side conditions.</p>

<p>Cook made this precise in 1978. Hoare logic cannot be complete outright, for
reasons that go back to Tarski rather than to engineering, but it is complete
<em>relative to an oracle</em>: complete if, mid-proof, you may consult the set of
true assertions and be told the answer. The standard history of the subject
describes the move as using the true assertions as an oracle “that can be
freely consulted in the correctness proof.”</p>

<p>So predicate logic is the subroutine and Hoare logic is the calling
convention, and everything hard about verifying a program is hard in the
assertion language rather than in the program logic wrapped around it. Fifty
years of tooling has been, in effect, fifty years of building better oracles:
decision procedures, SMT solvers, tactic languages, proof automation. The
calling convention never changed.</p>

<h2 id="two-bills-and-the-second-one-is-worse">Two bills, and the second one is worse</h2>

<p>The visible bill is proof effort, and everyone quotes it. Twenty person-years
against two. Seventy-six percent of the file. IronFleet worked hard to get its
implementation-layer ratio down to 3.6 lines of proof annotation per line of
executable code, which was a real achievement and is still 3.6.</p>

<p>The quieter bill is the specification, and it resists itemizing. Before any of
this begins, someone must write down what the system is supposed to do, in
logic, and nothing checks that statement. It is the axiom at the top.
IronFleet’s authors are direct about it: the trusted specification for IronRSL
is 85 lines and for IronKV is 34, “making them easy to inspect for
correctness.” Easy to inspect, by a person, reading it.</p>

<p>At the top of every verified system in the world there is a paragraph someone
read and believed.</p>

<p>seL4 draws the rest of its perimeter carefully: “We assume the correctness of
the compiler, assembly code, boot code, management of caches, and the
hardware; we prove everything else.” That is a clean line, drawn in public.
But the same paper contains a quieter sentence, about the kernel’s own
virtual-memory window: “We make this consistency argument only informally; our
model does not oblige us to prove it.” Even at the high-water mark of
machine-checked systems software, there is a seam where the proof stops and a
competent person says <em>and clearly</em>.</p>

<h2 id="verifications-are-not-messages">Verifications are not messages</h2>

<p>In 1979, De Millo, Lipton, and Perlis predicted that program verification was
bound to fail. They were wrong, and it is worth saying so first: seL4 exists,
CompCert ships inside avionics toolchains, and thirty years of work by people
who read their paper refutes the forecast.</p>

<p>What survives is the observation underneath. Mathematical proofs earn
confidence socially: they get read, restated, taught, doubted, patched, and
eventually absorbed. Program verifications cannot enter that process, because
of what they are:</p>

<blockquote>
  <p>Verifications are not messages; a person who ran out into the hall to
communicate his latest verification would rapidly find himself a social
pariah.</p>
</blockquote>

<p>Their blunter version: “Verifications are long and involved but shallow;
that’s what’s wrong with them.” A 200,000-line Isabelle script is a valuable
certificate. You can check it, and checking it is worth a great deal. You
cannot read it, disagree with one part of it, quote it in a design review, or
use it to show a new engineer why the kernel is safe. The knowledge exists and
does not circulate.</p>

<p>They saw the specification problem too, in a sentence that has not needed
revision in forty-five years: “The specifications for any reasonable compiler
or operating system fill volumes—and no one believes that they are
complete.”</p>

<h2 id="the-terms">The terms</h2>

<p><strong>Translate your intent into a formal assertion language, and a machine will
check everything downstream of that translation, with certainty.</strong></p>

<p>It is a good bargain, and the evidence for it is overwhelming. But notice what
you pay, because neither charge falls with scale. You must be able to make the
translation at all, which rules out every property you can state in English
and not in logic, and there are many. And what comes back is a certificate
rather than an argument a colleague can follow.</p>

<p>That is why verification concentrates where the stakes pay for it: kernels,
compilers, cryptography, flight control. Everywhere else, correctness
arguments still get made in English, constantly, and nothing checks them.</p>

<p>The next question is not how to make formal proofs cheaper. It is what happens
when the assertion language changes.</p>

<h2 id="references">References</h2>

<ul>
  <li>Robert W. Floyd. <a href="https://www.cs.tau.ac.il/~nachumd/term/FloydMeaning.pdf">Assigning Meanings to Programs</a>. Mathematical Aspects of Computer Science, Proc. Symposia in Applied Mathematics 19, AMS, 1967.</li>
  <li>C. A. R. Hoare. <a href="https://dl.acm.org/doi/10.1145/363235.363259">An Axiomatic Basis for Computer Programming</a>. Communications of the ACM 12(10), 1969. (Hoare writes the triple as P {Q} R, with the braces around the program; the modern form came later.)</li>
  <li>Stephen A. Cook. <a href="https://doi.org/10.1137/0207005">Soundness and Completeness of an Axiom System for Program Verification</a>. SIAM Journal on Computing 7(1), 1978.</li>
  <li>Krzysztof R. Apt and Ernst-Rüdiger Olderog. <a href="https://arxiv.org/abs/1904.03917">Fifty Years of Hoare’s Logic</a>. arXiv:1904.03917, 2019.</li>
  <li>Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, Thomas Sewell, Harvey Tuch, and Simon Winwood. <a href="https://trustworthy.systems/publications/nicta_full_text/1852.pdf">seL4: Formal Verification of an OS Kernel</a>. SOSP 2009.</li>
  <li>Xavier Leroy. <a href="https://xavierleroy.org/publi/compcert-CACM.pdf">Formal Verification of a Realistic Compiler</a>. Communications of the ACM 52(7), 2009.</li>
  <li>Chris Hawblitzel, Jon Howell, Manos Kapritsos, Jacob R. Lorch, Bryan Parno, Michael L. Roberts, Srinath Setty, and Brian Zill. <a href="https://sigops.org/s/conferences/sosp/2015/current/2015-Monterey/250-hawblitzel-online.pdf">IronFleet: Proving Practical Distributed Systems Correct</a>. SOSP 2015.</li>
  <li>Richard A. De Millo, Richard J. Lipton, and Alan J. Perlis. <a href="https://www.cs.umd.edu/~gasarch/BLOGPAPERS/social.pdf">Social Processes and Proofs of Theorems and Programs</a>. Communications of the ACM 22(5), 1979.</li>
</ul>]]></content><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><category term="systems" /><category term="research" /><category term="ai" /><summary type="html"><![CDATA[In 2009, NICTA finished a proof of the seL4 operating-system kernel. The kernel is 8,700 lines of C and 600 lines of assembler; its Isabelle proof is 200,000 lines. Building the kernel took about 2.2 person-years. Proving it took about twenty. Two years earlier, CompCert had done the same for a C compiler: 42,000 lines of Coq, now the Rocq Prover, of which 76% is the proof that compilation preserves what your program means. IronFleet later verified two distributed systems and reported a detail that deserves to be better known: both worked the first time they were run.]]></summary></entry><entry xml:lang="en"><title type="html">Token-Based Logic</title><link href="/naizhengtan_blog/posts/token-based-logic/" rel="alternate" type="text/html" title="Token-Based Logic" /><published>2026-09-20T13:00:00-04:00</published><updated>2026-09-20T13:00:00-04:00</updated><id>/naizhengtan_blog/posts/token-based-logic</id><content type="html" xml:base="/naizhengtan_blog/posts/token-based-logic/"><![CDATA[<p>Hoare logic leaves a deliberate gap. Its rule of consequence asks whether one
assertion implies another, and for fifty years the answer has come back from
formal logic and a decision procedure. Better solvers, better tactics, better
automation, and the gap kept the same shape.</p>

<p>Now imagine filling it with something else. Keep the triples, the invariants,
the composition rules. But write the assertions as sentences, and let the
judge of whether one sentence follows from another be a reader.</p>

<p>This is no longer a thought experiment. FM-Agent, out of Shanghai Jiao Tong,
“generalizes the inference rules of Hoare logic to operate over
natural-language specifications,” and it targets no prover at all. The paper
is blunt about the consequence: specifications it generates “cannot be used by
these tools, as they target formal specifications rather than natural language
specifications.” No Lean in the pipeline, no Rocq, no SMT. The oracle is a
language model.</p>

<p>That loses certainty. The question is what it buys.</p>

<h2 id="the-grain-matters-more-than-the-language">The grain matters more than the language</h2>

<p>The bad version of this idea is to hand a model a long argument and ask
whether it is correct. Long arguments invite superficial agreement. A reader
asked to approve sixty paragraphs will approve them.</p>

<p>So don’t ask that. Cut the argument into steps small enough that each one is a
single recognized inference, and check the steps.</p>

<p>Which inferences? There is no closed list, and pretending otherwise would be a
mistake, because human reasoning does not come in a finite alphabet. There is
deduction, induction, abduction; there is Bayesian updating, causal reasoning,
modal and temporal argument, analogy. None of this is a theory of how people
think.</p>

<p>The claim is narrower. A few of those patterns have been picked over long
enough that we know exactly when they hold, and the categorical syllogism is
the most worked-over of them. Two premises, one conclusion, three terms. All A
are B; all B are C; therefore all A are C. There are 256 syntactic forms, of
which 24 are traditionally valid, or 15 if you decline to assume the
categories are non-empty. That last clause is worth a moment: even in the most
scrutinized logic anyone ever wrote down, the count of valid forms turns on a
semantic convention you have to declare out loud. A system built on
natural-language inference will meet that kind of seam constantly, and the
seams are where it will break.</p>

<p>In practice the catalogue runs to a dozen entries or so. Barbara sits next to
modus ponens and modus tollens, disjunctive syllogism, conjunction and
simplification, case split, induction with its hypothesis, and, since the
subject is programs, Hoare’s rule of consequence and sequential composition.
It claims no completeness. What matters is that every entry has validity
conditions fixed in advance, so checking a step means asking whether it
instantiates a known shape, not whether it sounds convincing. Granularity is
the real design decision: a syllogism-sized step is small enough to judge and
large enough to be worth recording.</p>

<h2 id="one-move-four-costumes">One move, four costumes</h2>

<p>Small steps matter only if they compose, and the composition rule turns out to
be something we have been writing down, in different alphabets, for two
thousand years.</p>

<p><img src="/naizhengtan_blog/assets/img/topic15/fig1.svg" alt="The same inference appears as Aristotle's Barbara over terms, the chain rule over propositions, Gentzen's cut over sequents, and sequential composition in Hoare logic. In each, a middle element is produced by one premise, consumed by the next, and absent from the conclusion." /></p>

<p>In Barbara, the middle term connects subject and predicate and then vanishes
from the conclusion. In propositional logic, P implies Q and Q implies R yield
P implies R, and Q is gone. In Gentzen’s sequent calculus the move is called
cut, and the formula that disappears is named for it. In Hoare logic, a
program taking P to Q composes with one taking Q to R, where Q, the
intermediate state of the machine, is what the first delivers, what the second
requires, and what the composition does not mention.</p>

<p>Four kinds of object: classes, propositions, sequents, program states. One
structural move—the cut rule wearing four costumes, as I have taken to
calling it. This is the property that lets local checks add up. If every node
names its inputs and its output, and each can be judged on its own, then the
correctness of the whole argument becomes a property of the graph rather than
of anyone’s stamina in reading it end to end. The wiring is ordinary data
validation; no model needs to understand a single sentence to check it.</p>

<h2 id="what-you-are-trading">What you are trading</h2>

<p>Formal reasoning operates on truth values. Its judge is a decision procedure.
It fails by timing out, by meeting an undecidable fragment, or by being handed
a formula nobody can write. But when it says yes, it is right.</p>

<p>Token-based reasoning operates on sentences. Its judge is a reader, and it
fails by misreading: by being persuaded, by skimming, by agreeing with a
fluent step that does not follow. When it says yes, it is probably right, and
<em>probably</em> is not something you can prompt away.</p>

<p>Underneath the formal side there is a gradient worth naming. Syllogistic logic
is decidable, Boolean satisfiability is NP-complete, first-order predicate
logic is semi-decidable, and Hoare logic is undecidable outright—which is
why relative completeness is the ceiling. Each level buys expressive power
with tractability.</p>

<p>Token-based logic is not the next level on that ladder. It steps off it. There
is no complexity class for <em>a competent reader agrees</em>. What it buys is not a
larger decidable fragment but the ability to state the premise at all:
“the window never contains a repeated character,” “this cache is coherent
because a line is never dirty in two places,” “the retry is safe because the
operation is idempotent.” Real systems arguments are made of sentences like
these, and most of them have no formalization anyone is going to write.</p>

<h2 id="where-the-unchecked-step-lives">Where the unchecked step lives</h2>

<p>Formal verification is not unconditional either, and this part of the
comparison usually goes missing.</p>

<p>Its guarantee depends on a translation—from what you meant to what you wrote
in logic—and nothing checks that translation. It is the specification: at
the top of every verified system is a paragraph someone read and believed.
This is not hypothetical. Recent work on autoformalization keeps finding
formal and informal statements of the <em>same</em> problem disagreeing, in more than
half of the problems in miniF2F, a benchmark of short clean mathematical
statements and about as easy as translation ever gets.</p>

<p>So the question is not simply sound versus unsound. It is <strong>where you put the
unchecked natural-language step.</strong> Formal methods put it at the top, once, in
the specification, and check everything below it exactly. Token-based methods
spread it across every step and check each one approximately.</p>

<p>The first gives a strong guarantee about a statement that may be hard to
inspect. The second gives a weak guarantee about statements people can read.
Neither dominates. Which you want depends on whether the risk you are losing
sleep over is a subtle error below the specification or a mistaken idea above
it.</p>

<h2 id="relative-completeness-relative-to-a-reader">Relative completeness, relative to a reader</h2>

<p>Cook’s theorem makes Hoare logic complete relative to an oracle for the true
assertions—an oracle that cannot exist, for reasons older than computers,
and that we have spent fifty years approximating with solvers.</p>

<p>Token-based logic performs the same substitution with a different oracle: a
competent reader. That one cannot be trusted either, but for an entirely
different reason. It is not uncomputable; it is wrong sometimes. And unlike
the first, you can query it cheaply, about anything you can say, as often as
you like.</p>

<p>Which makes the next question empirical rather than philosophical. How often
does the reader fail, where, and can you build something that contains the
failures? That is what the last chapter is about.</p>

<h2 id="references">References</h2>

<ul>
  <li>C. A. R. Hoare. <a href="https://dl.acm.org/doi/10.1145/363235.363259">An Axiomatic Basis for Computer Programming</a>. Communications of the ACM 12(10), 1969.</li>
  <li>Gerhard Gentzen. Untersuchungen über das logische Schließen. Mathematische Zeitschrift 39, 1935. (The cut rule, and its elimination.)</li>
  <li>Stephen A. Cook. <a href="https://doi.org/10.1137/0207005">Soundness and Completeness of an Axiom System for Program Verification</a>. SIAM Journal on Computing 7(1), 1978.</li>
  <li>Azim Ospanov, Farzan Farnia, and Roozbeh Yousefzadeh. <a href="https://arxiv.org/abs/2511.03108">miniF2F-Lean Revisited: Reviewing Limitations and Charting a Path Forward</a>. arXiv:2511.03108, 2025.</li>
  <li>Haoran Ding, Zhaoguo Wang, and Haibo Chen. <a href="https://fm-agent.ai/">FM-Agent: Scaling Formal Methods to Large Systems via LLM-Based Hoare-Style Reasoning</a>. arXiv:2604.11556, 2026.</li>
</ul>]]></content><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><category term="systems" /><category term="research" /><category term="ai" /><summary type="html"><![CDATA[Hoare logic leaves a deliberate gap. Its rule of consequence asks whether one assertion implies another, and for fifty years the answer has come back from formal logic and a decision procedure. Better solvers, better tactics, better automation, and the gap kept the same shape.]]></summary></entry><entry xml:lang="en"><title type="html">Does It Work?</title><link href="/naizhengtan_blog/posts/does-it-work/" rel="alternate" type="text/html" title="Does It Work?" /><published>2026-09-20T12:00:00-04:00</published><updated>2026-09-20T12:00:00-04:00</updated><id>/naizhengtan_blog/posts/does-it-work</id><content type="html" xml:base="/naizhengtan_blog/posts/does-it-work/"><![CDATA[<p>Arguments about reasoning are cheap, so I built the smallest experiment that
could embarrass this one: take a fifteen-line algorithm, prove it twice (once
as a graph of natural-language inference steps, once mechanized in Rocq,
formerly Coq), and compare.</p>

<p>The algorithm is the sliding-window scan for the longest substring without
repeated characters. A left pointer, a running maximum, and a map from each
character to its last position. No cleverness anywhere, and one guard carrying
the entire correctness argument: if the current character was seen at or after
the left edge, advance the edge one past that earlier occurrence.</p>

<h2 id="two-proofs-of-the-same-fifteen-lines">Two proofs of the same fifteen lines</h2>

<p>The first proof has sixteen nodes, S1 through S16, each a small inference with
two or three premises, one conclusion, and a short justification. Edges are
typed by <em>how</em> a conclusion gets reused: as a direct premise, as the inductive
hypothesis, as one arm of a case split that rejoins by disjunction
elimination, as the base or step of an induction, or as one conjunct of the
final verdict. Structurally it is a Hoare proof—an invariant, a base case,
an inductive step with a two-way split on the last-occurrence lookup, and a
rule of consequence composing into the postcondition.</p>

<p><img src="/naizhengtan_blog/assets/img/topic15/fig2.svg" alt="The claim-level proof graph: sixteen syllogism-grade nodes. A base case and an inductive step, whose two cases rejoin at S11 by disjunction elimination, close the induction at S12; five independent lemmas then conjoin at the verdict. Edges are typed by how a conclusion is reused, which is what lets ordinary code check the wiring." /></p>

<p>The second proof establishes the same theorem in Rocq, with no axioms and
nothing left admitted. It runs 445 lines against the algorithm’s fifteen,
roughly twenty-nine to one, across 29 named lemmas, and took four
compile-debug cycles to land.</p>

<p>Three results came out of the comparison, and only one was expected.</p>

<p><strong>The graph made a good outline.</strong> The Rocq proof was written top-down from S1
through S16 in essentially one pass, and the invariant’s seven conjuncts are
the graph’s inductive-step nodes almost verbatim. The structure crossed the
language boundary at about zero cost, which is what you would predict if the
middle-term discipline really is indifferent to which level you work at.</p>

<p><strong>Formalization found no error in the graph.</strong> All four debug cycles were
tactic and bookkeeping work, not changes to the reasoning. Thin evidence, but
it suggests a disciplined graph can serve as a practical first draft of a
formal proof.</p>

<p><strong>The graph predicted the hard part.</strong> About 42% of the Rocq lines were index
and list bookkeeping no reviewer would ever doubt. The conceptual difficulty
sat where the graph said it would: maximality, the obligation that the left
pointer is <em>minimal</em> and not merely legal. Which suggests a workflow. Keep the
graph as the main artifact and formalize only the two or three high-risk
nodes.</p>

<p>One example, chosen by the person making the argument. I will come back to
that.</p>

<h2 id="the-planted-bug">The planted bug</h2>

<p>A proof you cannot break is not evidence, so I weakened the guard by one
character. The broken version advances the left edge only when the earlier
occurrence lies <em>strictly after</em> it, rather than at or after.</p>

<p>That fails when the earlier character sits exactly on the left edge: the
window keeps a duplicate and the reported length can be too large. On <code class="language-plaintext highlighter-rouge">"aa"</code>
the buggy program reports 2, though no duplicate-free substring of <code class="language-plaintext highlighter-rouge">"aa"</code> is
longer than one character.</p>

<p>The graph flags the guard node. Its justification had already named that
boundary as essential—before anyone touched the code, it said in as many
words that a strict comparison would fail to advance the pointer in exactly
this case. Then the counterexample stage stops arguing and runs the program.
Verdict: <code class="language-plaintext highlighter-rouge">REFUTED</code>, with a failed claim and an executed witness.</p>

<p>That is a localization result, not a soundness result, and localization is
what neither of the usual tools gives you. A test says something is wrong. A
machine-checked proof says nothing is wrong. A proof graph says <em>where</em>, in a
sentence a reviewer can read—which is precisely the property De Millo,
Lipton, and Perlis said verifications lack. It is a message.</p>

<h2 id="the-asymmetry">The asymmetry</h2>

<p>Working through the implementation, the thing that struck me hardest was not a
measurement. It was which stages need a model and which do not.</p>

<p>Producing the graph needs one, and so does judging whether a node’s premises
entail its conclusion. But checking that every premise reference resolves,
that each rule gets the arity it declares, that leaves bottom out only in the
precondition or the program’s own semantics: all of that is ordinary
deterministic code. The counterexample stage asks nobody’s opinion either. It
runs the program on inputs drawn from the precondition and compares against an
executable oracle.</p>

<p>Which gives the central asymmetry: <strong><code class="language-plaintext highlighter-rouge">REFUTED</code> is sound; <code class="language-plaintext highlighter-rouge">PROVED</code> is not.</strong> A
counterexample is a fact, because the program ran. A positive verdict is a
stack of judgments, and a stack of judgments is exactly as good as its worst
member. A useful system needs a third value, <code class="language-plaintext highlighter-rouge">INCONCLUSIVE</code>, and a rule that
it never silently passes.</p>

<p>That belongs in the headline rather than the caveats. It is the asymmetry that
makes tests useful and their absence meaningless, with one addition: here the
negative result arrives with an address.</p>

<h2 id="what-the-research-says">What the research says</h2>

<p>Two systems are attempting versions of this at a scale one example cannot
reach. Both come from IPADS at Shanghai Jiao Tong and share an author, so they
are one group’s bet rather than two groups converging.</p>

<p><strong>FM-Agent</strong> moves the checking step: 522 new bugs across four systems and
277,000 lines of code in about two days, with no solver in the loop. The
clever part is not the reasoning but the specification. FM-Agent derives each
function’s contract top-down from its <em>callers</em>, so a buggy implementation
cannot launder itself into its own spec. The one ablation is the number worth
remembering: specs derived from implementations find 57 bugs where specs
derived from callers find 339.</p>

<p>Those numbers need care. The 522 are reports that survived a filter; there is
no published false-positive rate, and for two of the four systems the filter’s
notion of expected behavior came from a machine-written specification. The
authors disclaim the right things: “we do not claim FM-Agent can replace
existing formal verification tools.”</p>

<p><strong>SpecFS</strong> is adjacent but different. A multi-part natural-language
specification, using pre- and postconditions and rely-guarantee contracts as a
writing discipline, generates about 4,300 lines of C across 45 modules; the
result fails 64 of 754 xfstests. Nothing is machine-checked: validation is
regression testing plus a second model reviewing the first. In one line,
FM-Agent replaces the prover with a language model and SpecFS replaces the
programmer with one, and only the first is a claim about verification.</p>

<p>The recipe is not new in mathematics either. Pseudo-formalization (decompose a
proof into modules, check each independently, aggregate) was published this
year and beats handing a judge the whole proof, flagging about
60% fewer false steps at matching recall. Its own reported limit is the
telling part: both verifiers still leave roughly half the known errors
uncaught. Breaking an argument up helps; it does not make the judge
trustworthy.</p>

<p>The benchmark picture is genuinely mixed. Frontier critics reach about 86.5%
step-level balanced accuracy on expert-annotated competition-math errors, and
verification rates consistently exceed solve rates—checking is easier than
producing, which is the economic premise of this whole approach and appears to
hold. But swap naturally occurring errors for adversarial ones (circular
reasoning, plausible-but-irrelevant steps, deliberate deception) and the best
score falls to 68.8% against a random floor of 50. Verdicts are unstable too:
judges flip between 25% and 71% of the time under simple pushback.</p>

<p>The worst result is the last. When a solution reaches the <em>right answer</em> by
invalid reasoning, frontier models score as low as 48% at catching it, below
chance on a binary question, while humans lose only about six points relative
to solving the problem themselves. The mechanism was measured rather than
guessed: an answer-confirmation bias, in which the model checks for the right
answer instead of checking each step. Patch the representation of the final
answer and the verdict flips.</p>

<p>That is exactly the shape of a node check: you hand the judge a conclusion and
ask whether the premises deliver it, and the judge can see the answer. The
response is architectural, not rhetorical: deterministic checks wherever they
are possible, judged steps small enough to sit inside the regime where
judgment is reliable, and never a bare pass.</p>

<h2 id="where-this-could-go">Where this could go</h2>

<p>The first use case is not where formal verification already thrives. It is the
ordinary argument in a design doc or a code review: why a retry is safe, why a
queue is bounded, why a cache stays coherent. Nobody is going to write a
Rocq specification for any of those. The alternative to a proof graph there is
not Lean; it is no checked argument at all.</p>

<p>Three more. <strong>Proof maintenance:</strong> the cost center of verified systems is not
the first proof but the second, after the code changes; a graph re-checks node
by node and reports which claims the change invalidated, which is what 200,000
lines of Isabelle cannot tell you kindly. <strong>Review by parts:</strong> a design
document that ships its argument as a graph can be disagreed with at node 7
rather than approved or rejected whole. <strong>Reviewing research:</strong> I have argued
elsewhere that our field’s
<a href="/naizhengtan_blog/posts/research-under-contention/">validation phase is the bottleneck</a>,
and part of why it is slow is that a paper’s argument is not checkable in
pieces.</p>

<h2 id="the-ledger">The ledger</h2>

<p>What this rests on, stated plainly: one hand-picked algorithm. A checker that
is itself a language model, with a measured 48% floor in the setting closest
to node checking. Prior work that already reports leaving half the errors
uncaught. And a system whose only defensible verdict is the negative one.</p>

<p>That last item reads like a deflation, and I want to end by refusing it. <em>This
step breaks, and this input proves it</em> is more than any tool offers today for
an argument written in English. It is local, it is readable, it is something
you can act on—and unlike 200,000 lines of certificate, you can say it out
loud in a hallway.</p>

<h2 id="references">References</h2>

<ul>
  <li>Haoran Ding, Zhaoguo Wang, and Haibo Chen. <a href="https://fm-agent.ai/">FM-Agent: Scaling Formal Methods to Large Systems via LLM-Based Hoare-Style Reasoning</a>. arXiv:2604.11556, 2026.</li>
  <li>Qingyuan Liu, Mo Zou, Hengbin Zhang, Dong Du, Yubin Xia, and Haibo Chen. <a href="https://llmnativeos.github.io/specfs/">Sharpen the Spec, Cut the Code: A Case for Generative File System with SysSpec</a>. FAST 2026.</li>
  <li>Slim Barkallah, Luke Bailey, Kaiyue Wen, Mohammed Abouzaid, and Tengyu Ma. <a href="https://arxiv.org/abs/2605.20531">Pseudo-Formalization for Automatic Proof Verification</a>. arXiv:2605.20531, 2026.</li>
  <li>Shrey Pandit, Austin Xu, Xuan-Phi Nguyen, Yifei Ming, Caiming Xiong, and Shafiq Joty. <a href="https://arxiv.org/abs/2510.13744">Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math</a>. arXiv:2510.13744, 2025.</li>
  <li>Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. <a href="https://arxiv.org/abs/2412.06559">ProcessBench: Identifying Process Errors in Mathematical Reasoning</a>. ACL 2025.</li>
  <li>Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, and Yu Cheng. <a href="https://arxiv.org/abs/2501.03124">PRMBench: A Fine-Grained and Challenging Benchmark for Process-Level Reward Models</a>. ACL 2025.</li>
  <li>Mingzhong Sun, Teresa Yeo, Armando Solar-Lezama, and Tan Zhi-Xuan. <a href="https://arxiv.org/abs/2606.01462">An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models</a>. arXiv:2606.01462, 2026.</li>
  <li>Justin Zhao, Himaghna Bhattacharjee, Hannah Korevaar, Bhaktipriya Radharapu, and Khalid El-Arini. <a href="https://arxiv.org/abs/2608.12645">Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistence</a>. arXiv:2608.12645, 2026.</li>
  <li>Sergiu Bursuc, Theodore Ehrenborg, Shaowei Lin, Lacramioara Astefanoaei, Ionel Emilian Chiosa, Jure Kukovec, Alok Singh, Oliver Butterley, Adem Bizid, Quinn Dougherty, Miranda Zhao, Max Tan, and Max Tegmark. <a href="https://arxiv.org/abs/2509.22908">A Benchmark for Vericoding: Formally Verified Program Synthesis</a>. arXiv:2509.22908, 2025.</li>
  <li>Richard A. De Millo, Richard J. Lipton, and Alan J. Perlis. <a href="https://www.cs.umd.edu/~gasarch/BLOGPAPERS/social.pdf">Social Processes and Proofs of Theorems and Programs</a>. Communications of the ACM 22(5), 1979.</li>
</ul>]]></content><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><category term="systems" /><category term="research" /><category term="ai" /><summary type="html"><![CDATA[Arguments about reasoning are cheap, so I built the smallest experiment that could embarrass this one: take a fifteen-line algorithm, prove it twice (once as a graph of natural-language inference steps, once mechanized in Rocq, formerly Coq), and compare.]]></summary></entry><entry xml:lang="en"><title type="html">Replay Comes Back</title><link href="/naizhengtan_blog/posts/replay-comes-back/" rel="alternate" type="text/html" title="Replay Comes Back" /><published>2026-09-08T01:00:00-04:00</published><updated>2026-09-08T01:00:00-04:00</updated><id>/naizhengtan_blog/posts/replay-comes-back</id><content type="html" xml:base="/naizhengtan_blog/posts/replay-comes-back/"><![CDATA[<p>An agent ran a task yesterday and it worked. It ran the same task today and
did something else. Which of the two runs should you trust, and what exactly
was different?</p>

<p>Right now nobody can answer that. The best practice is: run it again and
hope. That was tolerable when models wrote poems, and it is not tolerable
now that agents hold shell access. In our own measurements across 17 models
and 10 system-administration tasks, pass rates routinely exceed 80%, yet two
runs of the same task agree on the sequence of state-modifying actions only
about 39% of the time. We have also watched runs diverge on nothing more
than a clock: the same task, the same model, a different timestamp visible
in the environment, and the agent takes another path.</p>

<p>This is not new to systems people. It is called non-determinism, and the
systems community has been grappling with it for decades. What is different
today is that the AI stack amplifies non-determinism across many layers.</p>

<h2 id="three-layers-of-non-determinism">Three layers of non-determinism</h2>

<p>Unlike in traditional systems, where non-determinism is often contained
within an abstraction layer, in AI stacks it can silently propagate upward
and affect higher layers.</p>

<p><img src="/naizhengtan_blog/assets/img/topic14/fig1.svg" alt="Non-determinism enters an agent run at three layers---tensor, token, and trace. Record every value the run could not have predicted, and you can replay it, free one layer to find the culprit, or change one value and watch the path fork." /></p>

<p><strong>Tensor.</strong> The arithmetic itself. Most inference kernels lack <em>batch
invariance</em>: a reduction is carried out differently depending on how many
requests are batched together, so a request’s output depends on the server’s
load at the moment it arrived. Batch-invariant kernels remove this
variation, at the cost of roughly 1.6x the latency. Underneath sit the
slower drifts—a different GPU, a different library version—which vendors
are explicit about: cuBLAS promises bitwise reproducibility only on the same
architecture with the same number of SMs, and never across toolkit versions.</p>

<p><strong>Token.</strong> Sampling. Temperature, top-p, and the RNG that draws from the
distribution. This layer is the one everybody knows about and the only one
with a user-facing knob, and even that knob is weak: the seed parameter is
documented as a best-effort promise that determinism is “not guaranteed,”
and it is now being deprecated.</p>

<p><strong>Trace.</strong> The world. Timestamps, PIDs, file listings, network responses,
tool output, and whatever another process did to the machine between two
runs. This layer has a property the other two do not: the agent <em>changes</em>
it. The run modifies the state that the next step reads.</p>

<p>Moreover, the layers compose upward: a tie broken differently in a reduction
becomes a different token, which becomes a different shell command, which
becomes a different sequence of actions. Noise at the bottom is behavior at
the top. That is why a debugger for agents cannot live at the framework
boundary.</p>

<h2 id="tttr-record-replay-and-steer">TTTR: record, replay, and steer</h2>

<p><strong>TTTR is Tensor-Token-Trace-level Replay</strong>: record a run’s non-determinism
at all three layers, and replay it.</p>

<p><em>Record.</em> Capture every value the run could not have predicted—the batch
composition or the kernel outputs at the tensor layer, the sampled tokens
and seeds at the token layer, the clock, PIDs, and tool results at the trace
layer. This is an old idea with a long systems pedigree: ReVirt logged
beneath an untrusted OS, R2 let the programmer choose where to cut, and rr
made record-replay cheap enough to use every day.</p>

<p><em>Replay.</em> Run it back and get the same execution. Here the classic
literature supplies a warning worth taking: ODR gave up on reproducing the
identical execution and guaranteed only the same <em>output</em>, because inferring
an exact schedule is ruinously expensive. For agents, bit-identical replay
is likewise the wrong bar. What you want is a replay that reaches the same
decisions.</p>

<p><em>Selective replay.</em> This is the point of the whole exercise. Pin two layers
and let the third float. Or replay everything faithfully except one recorded
value—advance the clock by a second, hand back a different tool result,
flip one sampled token—and watch where the trajectory forks. The systems
precedent is mutable replay, which replays a recording against modified
code; TTTR generalizes it from code mutation to any single source of
non-determinism. And it turns delta debugging—Zeller’s method of
binary-searching the minimal failure-inducing change, from the paper titled
“Yesterday, my Program Worked. Today, it Does Not. Why?”—into something
you can run <em>across layers</em> rather than within one input.</p>

<h2 id="what-you-would-do-with-it">What you would do with it</h2>

<p><strong>Locate the layer.</strong> The agent worked yesterday and fails today. Replay
today’s run with the tensor layer pinned to yesterday’s recorded values: if
the failure disappears, the arithmetic moved under you. Pin the trace layer
instead: if it disappears, the environment changed. This is triage that
currently does not exist, and it answers the question every on-call engineer
actually has—not “what did the agent do” but “what is different.”</p>

<p><strong>Run the counterfactual.</strong> Change exactly one thing and hold everything
else. Would the agent still have killed the process if the port had been
free? Does this task depend on the date? Today those questions are answered
with anecdotes and re-runs, which confound every variable at once. Replay
converts them into controlled experiments with a single independent
variable. A recent preprint on causal attribution for agent failures does
this statistically, by intervening on a step and re-sampling; a replay
system would let you do it mechanically, which is both cheaper and sharper.</p>

<p><strong>Report honest numbers.</strong> Pass rate hides everything this post is about.
With replay you can hold the lower layers fixed and measure how much of a
benchmark difference is the model and how much is noise—a variance
decomposition for agent evaluation. A benchmark delta of two points means
nothing if two runs of the same agent agree only 39% of the time on what
they did.</p>

<h2 id="determinism-was-always-engineered">Determinism was always engineered</h2>

<p>Determinism was never a property computers had naturally. It was built, at
considerable cost, and then quietly relied on by everyone who ever attached
a debugger. Threads eroded it, and the record-replay literature spent
decades winning it back. Agents have now removed it again.</p>

<p>We are on our way to recovering determinism in AI. This time the payoff
reaches past debugging: the group that built batch-invariant kernels found
that once inference was deterministic their reinforcement learning became
genuinely on-policy, the divergence between sampler and trainer sitting flat
at zero. A system you can replay is a system you can <a href="/naizhengtan_blog/posts/consistency-checking-is-the-new-verifier/">check</a>, and a
system you can check is one you can train against. We gave up
reproducibility at precisely the moment we started running programs whose
behavior we cannot predict.</p>

<h2 id="references">References</h2>

<ul>
  <li>
    <p>Tian Lu, Zikai Wang, and Cheng Tan.
<a href="https://sccp-workshop.github.io/sccp26/papers/sccp26-behavioral-consistency-llm-agents.pdf">Beyond Pass Rate: A Hierarchy of Behavioral Consistency for LLM Agents</a>.
SCCP 2026.</p>
  </li>
  <li>
    <p>Horace He and Thinking Machines Lab.
<a href="https://thinkingmachines.ai/blog/defeating-non-determinism-in-llm-inference/">Defeating Non-determinism in LLM Inference</a>.
Thinking Machines Lab: Connectionism, September 2025.</p>
  </li>
  <li>
    <p>NVIDIA.
<a href="https://docs.nvidia.com/cuda/cublas/index.html#results-reproducibility">cuBLAS: results reproducibility</a>
and <a href="https://docs.nvidia.com/cuda/floating-point/index.html">Floating Point and IEEE 754 Compliance for NVIDIA GPUs</a>.</p>
  </li>
  <li>
    <p>George W. Dunlap, Samuel T. King, Sukru Cinar, Murtaza A. Basrai, and
Peter M. Chen.
<a href="https://www.usenix.org/conference/osdi-02/revirt-enabling-intrusion-analysis-through-virtual-machine-logging-and-replay">ReVirt: Enabling Intrusion Analysis through Virtual-Machine Logging and Replay</a>.
OSDI 2002.</p>
  </li>
  <li>
    <p>Zhenyu Guo, Xi Wang, Jian Tang, Xuezheng Liu, Zhilei Xu, Ming Wu,
M. Frans Kaashoek, and Zheng Zhang.
<a href="https://www.usenix.org/legacy/event/osdi08/tech/full_papers/guo/guo.pdf">R2: An Application-Level Kernel for Record and Replay</a>.
OSDI 2008.</p>
  </li>
  <li>
    <p>Gautam Altekar and Ion Stoica.
<a href="https://www.sigops.org/s/conferences/sosp/2009/papers/altekar-sosp09.pdf">ODR: Output-Deterministic Replay for Multicore Debugging</a>.
SOSP 2009.</p>
  </li>
  <li>
    <p>Nicolas Viennot, Siddharth Nair, and Jason Nieh.
<a href="https://dl.acm.org/doi/10.1145/2499368.2451130">Transparent Mutable Replay for Multicore Debugging and Patch Validation</a>.
ASPLOS 2013.</p>
  </li>
  <li>
    <p>Robert O’Callahan, Chris Jones, Nathan Froyd, Kyle Huey, Albert Noll, and
Nimrod Partush.
<a href="https://www.usenix.org/conference/atc17/technical-sessions/presentation/ocallahan">Engineering Record And Replay For Deployability</a>.
USENIX ATC 2017.</p>
  </li>
  <li>
    <p>Andreas Zeller.
<a href="https://www.st.cs.uni-saarland.de/publications/details/zeller-esec-1999/">Yesterday, my Program Worked. Today, it Does Not. Why?</a>
ESEC/FSE 1999. See also Zeller and Hildebrandt,
<a href="https://dl.acm.org/doi/10.1109/32.988498">Simplifying and Isolating Failure-Inducing Input</a>,
IEEE TSE 28(2), 2002.</p>
  </li>
  <li>
    <p>Jaineet Shah.
<a href="https://arxiv.org/abs/2606.08275">Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures</a>.
arXiv:2606.08275, 2026.</p>
  </li>
</ul>]]></content><author><name>Cheng Tan</name><email>c.tan@northeastern.edu</email></author><category term="systems" /><category term="ai" /><category term="verification" /><summary type="html"><![CDATA[An agent ran a task yesterday and it worked. It ran the same task today and did something else. Which of the two runs should you trust, and what exactly was different?]]></summary></entry></feed>