Cheng Tan

Research Under Contention

Peer review is the validation phase of an optimistic protocol—and that protocol is thrashing

· systemsresearchai ·中文

This year I served on the program committees of several major systems conferences, and for roughly half of the papers in my AI-systems pile, I could not honestly say that I reviewed them as well as I would have a few years ago. Thinking about why led me somewhere I did not expect: something fundamental about paper reviewing has changed, perhaps permanently. What follows is my attempt to name the mechanism.

Research runs optimistic concurrency control

Treat all human knowledge as one database. A research project is a transaction: read a snapshot of the literature, compute for months or years in a private workspace, then submit the result for validation. Peer review is the validation step—does this work conflict with what is already committed, and was what it read still current? Pass, and the result commits: published, visible, safe for every later transaction to build on. A wrong review is a bad commit: incorrect data written, stale numbers blessed, or someone else’s correct data quietly polluted downstream.

This is not a loose analogy. It is, phase for phase, optimistic concurrency control, the protocol written down in 1981: a read phase, a validation phase, and, if validation succeeds, a write phase. It is called optimistic because nobody locks anything. Everyone computes in parallel, “‘hoping’ that conflicts between transactions will not occur”—the scare quotes around hoping belong to the original paper. And when validation fails, the transaction “will be backed up and start over again as a new transaction,” which every rejected author knows by its other name: the next deadline.

Science could afford this optimism for three centuries because science was a low-contention workload. Few transactions touched the same rows, they ran for years, and they rarely collided. The commit phase was slow, human, and adequate. I have spent a good part of my career building checkers that decide whether concurrent transactions serialize; the joke is on me. The transactions are us.

The workload changed

Databases measured what happens next within a few years of the protocol: past a certain load, optimistic schemes “start thrashing”—work is wasted, transactions restart, and the restarts themselves add load. That is our conference calendar. Every rejected paper re-enters the queue at the next deadline, and the queue keeps growing: OSDI ‘26 received more than twice as many submissions as any previous year, and went multi-track for the first time.

Contention rose because the transactions got shorter and the rows got hotter. Multiple independent discovery was always the norm, but it used to be settled at leisure, years apart. Now that execution is cheap, the multiples land months apart, inside the same reviewing cycle—and many of them sit on arXiv, readable by every other in-flight transaction long before any committee has validated them. Databases have a name for reading data that was never committed: a dirty read.

Worse, the validator itself has stopped returning a reliable answer. It was noisy even in calm times: the ML community ran the experiment twice, in 2014 and 2021, and both times roughly half the papers accepted by one committee would have been rejected by an independent second one. Now part of the validator is a machine—by one estimate, between 6.5 and 16.9 percent of recent ML conference reviews were substantially modified by LLMs. A validator that guesses commits corrupt data, and corrupt data does not sit quietly: a wrong number, once committed, becomes the baseline that the next hundred honest transactions must beat.

What publication becomes

Optimism was never a strategy; it was a workload assumption. It held for three hundred years and it does not hold now. When that happens to a database, you do not abandon the database. You rethink the protocol.

Databases spent forty years developing answers to contention, and the interesting part is that our field is already adopting them—one at a time, without naming them, and without ever deciding to.

  • Locking: plant the arXiv flag the moment the idea exists, before the artifact does.
  • Partitioning: shard into micro-communities where twenty experts validate one another, each shard slowly developing its own truths.
  • Read-uncommitted: cite the preprint, because the citation graph now runs months ahead of validation.
  • Eventual consistency: let social media carry the gossip protocol, and let the conference become an archival ceremony for results the field absorbed last spring.

Each of these is individually rational. Each gives away the exact guarantee the commit phase existed to provide—that committed knowledge can be built on without re-verifying it yourself.

That is the trade-off, and databases have never found a way around it: weaker consistency buys throughput. We can commit faster, or we can commit with stronger guarantees, but not both. The field is currently choosing speed, one workaround at a time.

If that is the choice, we should at least make it with open eyes, because the bill is already itemized. Split lineage: two versions of a result, both public, neither reconciled. Duplicate commits: concurrent papers implementing the same idea, each honest, neither wrong. A literature that is fast, current, and no longer serializable.

I do not know which isolation level science should run at. I do know it has become a systems question—and that the level we drift into by default is rarely the one we would have chosen.

References