Keep It Close

Third in the series — Nothing New Under the Sun — on locality, the oldest performance trick, and the consistency bill it always sends

This is the third entry in a series on ideas that computer science keeps rediscovering under new names, after deduplication and search. This one is older than both, and it sounds too simple to be a principle: keep the data close to where it is used. Most of performance engineering, I’ve come to believe, is this one idea applied at a new distance.

Performance is mostly geography

The reason is physics. A signal cannot cross a wire faster than light, and in one cycle of a 3 GHz clock light travels about ten centimeters. Real on-chip wires are much slower than that, because their delay grows with the square of their length. So distance is time. It is also energy: on a 45 nm chip, a 32-bit floating-point add costs about 0.9 picojoules, and reading the same 32 bits from DRAM costs about 640. Arithmetic is cheap. Moving the operands to the arithmetic is what you pay for.

Programs are kind to us here. They tend to reuse what they just touched and to touch what sits next to it. Peter Denning named this locality in the late 1960s, when virtual-memory systems started thrashing, and his working-set model explained why: a program needs only a small, slowly changing set of pages at any moment, so keep that set close and the rest can live far away.

The same move then reappears at every scale. Caches keep recently used memory next to the core. In the 1980s, Lisp machines built hardware for a language made of pointers: CDR-coding stored a list’s cells next to each other, and their garbage collectors copied related objects onto the same page to keep data local. By the early 1990s they had lost to general-purpose workstations that ran Lisp as fast or faster.

AI kernels are the newest instance. FlashAttention did not change the attention computation. It changed where the intermediate result lives: instead of writing the N×N attention matrix to GPU memory and reading it back, it computes in tiles that stay in on-chip SRAM. GPT-2 trained three times faster. Kernel fusion in general is the same trade: the same arithmetic, fewer trips.

None of these change what is computed. They change where the data is when it is computed.

Every copy is a promise

How do you get data close? You copy it. A cache line is a copy of a memory block. A replica is a copy of a database. A page in a buffer pool is a copy of a page on disk. And the moment there are two copies, they can disagree.

So the second wave follows the first, always. Multiprocessors got private caches and then needed coherence protocols; the MESI states date from 1984. Replicated services needed consistency, and linearizability gave it a definition in 1990. Databases that let transactions read their own snapshots needed isolation levels, and a long argument about which anomalies are acceptable. Different names, one problem: locality is copying, and consistency is the bill.

The bill is not always due, and when it is not tells you something. FlashAttention pays nothing for its tiles, because nobody else ever reads them. They are private and short-lived. A read-only copy is also free: a thousand caches can hold the same constant without a protocol. The bill arrives exactly when data is copied, shared, and written. That gives a rule simple enough to check a design against: copy what nobody writes; be careful about copying anything that someone else writes.

Every level of the memory hierarchy buys speed with a copy, and every shared copy needs a protocol to keep it agreeing with the others. A private copy, like a fused kernel's tile, needs none. Latencies are approximate.

Moving the data away on purpose

The cloud has spent the last decade doing the opposite: moving data away from compute. Amazon Aurora separates the database engine from its storage; only redo log records cross the network, and the storage keeps six copies across three availability zones. CXL memory pools pull DRAM out of individual servers, because up to a quarter of it sits stranded and memory can be half of a server’s cost. The price is distance: Pond, Microsoft’s CXL pool design, measures 70 to 90 nanoseconds added to each access.

Why give up locality? For utilization, elasticity, and failure independence, which locality never offered. And one far-away source of truth is, for a while, simpler to keep consistent than many nearby copies.

The while does not last. Aurora’s compute nodes still keep a buffer cache, so most reads never leave the machine. CXL 3.0 lets hosts share pooled memory coherently, which means the pool can reach back and invalidate a host’s cached lines. Disaggregate, and the latency pushes you to cache; cache, and the copies send their bill. The pendulum swings between locality and a single truth, and each swing reinvents the protocol of the last, at a larger distance.

This is where a piece of my own work lives. When the protocols that keep copies agreeing become too complex to trust, you can check the outcome instead: record what clients observed and ask whether some serial order explains it. That is consistency checking, and it exists because locality keeps making copies.

The context window is a cache

Now look at an AI agent. Its context window is the fastest memory it has, and what it puts there is a copy: the file it read ten minutes ago, the test output from the last run, the design it was told about this morning. Choosing what goes in context is choosing a working set. Agents that retrieve well are agents with good locality.

And their copies go stale. Another agent edits the file, a person changes the plan, and the agent keeps acting on its own copy. With one agent, this is annoying. With many agents sharing a repository, it is a coherence problem with no coherence protocol. Re-reading a file before editing it is invalidation by hand. I expect multi-agent systems to rediscover, in some form, the whole second wave: ownership, invalidation, and an argument about which anomalies are acceptable.

Close, and agreed

Keep it close is half of the rule. The other half is to know who else holds a copy. Locality made computers fast, and consistency is what it costs to share what you made fast. I don’t expect either half to go away. I expect the next system, at whatever distance it works, to pay for both again.

References