Cheng Tan

Every Decision Has a Binding Time

Four levels of systems decision-making, and why only compiled code can live at the bottom one

· systemsairesearch ·中文

Take any behavior of a running system and ask one question about it: when was this decided? Not who decided it, not why—when did it stop being open and become fixed? Sort every decision in a system by the answer and something tidy happens. They don’t spread evenly across time. They clump.

Four levels of systems decision-making

L1, design. The architecture, the objectives it’s built to serve, the invariants it must not violate. Decided once, early, usually in an argument between a few people, and thereafter treated as terrain rather than choice. Log-structured or update-in-place. One tier or three. Whether stale reads are permitted at all. These decisions are made on the timescale of a project and revisited on the timescale of a rewrite.

L2, implementation. The actual algorithm and the code that expresses it. LRU or CLOCK or something else. The exact eviction rule, the branch order, the data layout. Decided when someone writes it, changed on the timescale of a code review, and frozen the moment the compiler runs.

L3, policy parameters. The tunables: buffer sizes, thresholds, timeouts, batch sizes, weights. Decided at configuration time, changed on the timescale of a deploy or a knob-turn—minutes to weeks. Model weights live here too, which is a point I’ll come back to.

L4, runtime decisions. The per-event choices. Evict this line. Admit this packet. Run this thread next. Decided fresh, billions of times a second, and consumed instantly—the decision has no life beyond the event that prompted it.

The structure is nested, and that’s the part that makes it more than a list: each level defines the space the next one chooses from. The design determines which algorithms are even expressible. The implementation determines which parameters exist to be set. The parameters determine which runtime choices are reachable. You cannot make a decision at L3 that L1 did not leave room for. When people say a system is “flexible,” they almost always mean its upper levels left unusually large spaces below them.

Between L1 and L2, the seam even has a name every programmer already uses: the interface. L1 defines the interface; L2 decides how to implement it. You can swap LRU for CLOCK without renegotiating the architecture precisely because the architecture only ever spoke for the interface, never for what happens behind it.

The gradient

Descend the levels and two quantities move in opposite directions.

Frequency rises by orders of magnitude at every step—from a handful of design decisions in a system’s lifetime, to thousands of lines of code, to hundreds of knobs, to billions of events per second. And consequence per decision falls just as fast. Get the architecture wrong and you rewrite the system. Get one eviction wrong and you take a cache miss.

This inverse relationship is not a coincidence; it’s what makes the levels stable rather than arbitrary. A decision you must make constantly cannot be allowed to matter much individually, or the system would be impossible to operate. And a decision that matters enormously had better be rare, or nobody could afford the deliberation it deserves. Systems are, among other things, machines for pushing consequence upward and frequency downward.

Which gives the central technical fact of this series. Only compiled code can occupy L4. Not as a matter of taste—as arithmetic. A decision made a hundred million times a second has a budget measured in nanoseconds, roughly the cost of a handful of instructions and one cache miss you were hoping to avoid. Anything that thinks about the decision—a forward pass, a lookup into a learned structure, a call into a runtime that isn’t already hot in i-cache—costs microseconds, three orders of magnitude more than the decision is worth. You can shrink the model until it fits, but by then you have built a lookup table with extra steps, and you should ask why you didn’t just write the lookup table.

Production practice, on the rare occasion a model is admitted at runtime, confirms the arithmetic rather than defying it. The learned lifetime predictor inside Google’s VM scheduler serves a decision made tens of times a second—not billions—and even at that gentle cadence it was compiled into the scheduler’s own binary, canaried and rolled out with it, and budgeted at nine microseconds. The closer a model gets to L4, the more it is forced to behave like what already lives there: code.

So the correct place for learning to live is not L4. It’s L2 and L3—in the code, and in the constants that code reads. Learning about L4 behavior, certainly. Learning at L4, almost never.

L0: the level that isn’t a level

The hierarchy as stated has a hole at the top. L1 is design, and design serves objectives—but where do the objectives come from?

Call it L0: purpose. What the system is for. What “better” means. The service-level agreement, the thing you are willing to trade for the thing you aren’t, the failure you’d accept versus the one that ends the business. It is not a design decision; it’s the thing design decisions are answerable to.

You need this level explicitly, and not for sentimental reasons. Without it, the sentence “the system optimizes itself” has no referent. Optimizes what? Every level below L0 is an optimization problem, which means every level below L0 has an objective handed down from above. L0 is where the regress stops. It’s the only level in the stack with nothing above it to appeal to—which is exactly why it’s the one that stays human, a point the last post in this series is really about.

Two axes, or the map is useless

Here’s the problem with everything I’ve written so far: it describes a system with nobody in it.

Levels tell you when decisions get frozen. They say nothing about who does the freezing, or under what discipline. Two systems can have identical level structure and be nothing alike—one where a human hand-tunes every parameter after reading a dashboard, one where a search process rewrites the same parameters hourly. The hierarchy can’t tell them apart. So it needs two orthogonal axes.

Allocation—who decides at each point. Call it α: a map from decision points to deciders. Human, algorithm, model, agent, or some mixture. Allocation is not a property of a level; it’s a property of a system, and it is the thing that actually changes when a system becomes more automated. Most arguments about AI in systems are arguments about α that never say so, which is why they go badly—one person is talking about who writes the eviction policy and the other is talking about who sets the SLA, and both think they’re disagreeing about “AI.”

Process—under what discipline. Call it ρ: how a decision is validated before it takes effect, what gates it must pass, what happens when it’s wrong, and how fast you can undo it. Canaries, shadow traffic, staged rollout, a kill switch, a fallback path that is known-good and always warm.

These two are independent, and that independence is the useful part. You can hand a decision to a machine and keep a brutal validation process, or keep a decision human and have no process at all—which is most production systems, honestly, and is why “a human decided it” is weaker assurance than it sounds. As you move α toward the machine, ρ has to carry proportionally more weight, because the thing you gave up was a slow deliberate creature who would probably have noticed. Automation doesn’t remove the need for judgment. It relocates it into the gate.

What I’m borrowing

Almost nothing above is new, and the honest thing is to say what’s inherited before someone else does.

Binding time is not my term. It’s a standard concept in programming languages, and the textbook definition is almost embarrassingly on the nose: the point at which a binding is created or, more generally, the point at which any implementation decision is made.

And the autonomy part is older than the AI conversation. In 2001 IBM launched autonomic computing, and asked for systems that manage themselves under delegated policies. It got a control loop that monitors, analyzes, plans, and executes over shared knowledge, and a five-level maturity model. The levels are keyed to which stage of the loop the human still owns: at the bottom, people do the analyzing and the planning; predictive systems analyze and recommend while people approve; only at the top does the loop close by itself. That is the α axis, twenty years early, and I’d rather stand on it than rediscover it.

That’s the coordinate system. The next post uses it to say what “AI-native” should mean, and what it can’t be allowed to mean.

References