Cheng Tan

Learning at the Wrong Time

Machine learning for systems did reach production—but only after being compiled, distilled, or rewritten into ordinary code. That pattern is the whole story

· systemsairesearch ·中文

Let me start by giving the field its due, because the criticism only counts if the praise is real.

Machine learning for systems has been one of the more intellectually satisfying lines of work in computer systems over the last decade—and I say that as a participant, not a spectator. Our verified neural networks have served as a standard VNN-Comp benchmark for five consecutive years; we built the first system that trains fully verified neural networks for systems; one of our trained models has been experimentally deployed at Azure.

The founding move was lovely: a database index is really just a function from key to position, so learn the function. Once you see it, the pattern generalizes everywhere, because systems are full of predictors we wrote by hand. A cache eviction policy is a prediction about future reuse. A scheduler is a prediction about job length. Congestion control is a prediction about the network. Prefetchers, branch predictors, query optimizers, autoscalers—all of them are guesses, all of them hand-tuned by people over decades, and all of them, in principle, learnable from data.

The tempting next sentence is that none of it shipped. I believed a version of that myself, and it’s wrong. Plenty of it shipped. What’s interesting is the shape of what shipped.

Look at what’s actually running

Azure’s VM scheduler consults a learned model of workload behavior—random forests and gradient-boosted trees, a few hundred kilobytes, trained offline and served from a library linked into the client, which returns an explicit “no prediction” when it doesn’t recognize the situation. Google’s flash cache admission policy was hand-tuned for years and is now set by an optimizer that partitions traffic into categories and solves a knapsack problem, worth a 6.5% cut in operational cost. Microsoft’s SCOPE query optimizer is steered by a model trained in what its authors call a massive offline pipeline, enabled by default in production.

Even the two most interesting cases obey the pattern. Google’s VM scheduler does consult a learned lifetime model in production—but the model is compiled into the Borg binary itself, canaried and rolled out like any other code, answering in nine microseconds at a decision point that fires tens of times a second, not billions. And the heuristic that packs Google’s fleet, in production over a year and recovering about 0.7% of its worldwide compute, is four lines of arithmetic over two ratios, produced by an offline search.

Now hold those next to the papers. The learned index that beat B-trees, the deep reinforcement-learning scheduler, the neural congestion controller. What went to production is a random forest behind a fallback, a knapsack solver, four lines of arithmetic, and one runtime model that had to be welded into the scheduler binary to be allowed in. The learning is real and the deployments are real. The neural network on the hot path is what didn’t survive.

So the question isn’t why ML failed in systems. It’s why everything that succeeded arrived wearing the same disguise.

Five barriers

Here are the five barriers—widely agreed on, I think—to shipping the model itself in production.

Inference costs too much. The decisions we most want to improve are the ones made most often, and the ones made most often have nanosecond budgets. Many systems cannot afford even one round of CPU–GPU communication in an online decision, to say nothing of the forward pass waiting on the other side of it.

It degrades silently. A learned component trained on last quarter’s workload doesn’t fail when the workload moves. It gets quietly worse. We saw this ourselves in Cartur, our project that places Azure VMs across NUMA nodes: over time, the clients’ patterns simply shift out from under the model, and nothing announces the shift.

The tail is wrong. A learned policy better on average can be much worse in the worst case, and production systems are graded on the worst case. The theory here is unforgiving. A classical Bloom filter’s false-positive rate is the same for any query set; a learned one’s depends on which queries you happen to ask, and no adversary is required to break it—queries merely resembling the stored keys are enough to move the empirical error rate by 5×.

Maintainability. The cost comes from two sides. Shipping a model means shipping training-data collection, feature extraction, retraining, versioning, drift monitoring, and a rollback story—the classic paper on this draws the learning code as a small box adrift in a much larger diagram of infrastructure. You didn’t add a component; you adopted a dependent. And when the dependent misbehaves, you cannot read it. A heuristic that goes wrong points you at a branch; a model that goes wrong points you at nothing, because the behavior lives in the weights, and the weights don’t explain themselves.

The experts are expensive. Doing this well requires fluency in both machine learning and systems. That intersection is thin, and a technique requiring a rare kind of person to operate is a technique that doesn’t scale.

Four of them are one of them

Look at the list again, because it isn’t five problems.

Inference overhead exists because the model runs at runtime. Silent degradation exists because the model runs at runtime, so nothing between training and production ever re-examines it. The tail problem exists because the model runs at runtime—you can’t bound what you can’t inspect ahead of time. The maintenance burden exists because the model runs at runtime, so the machinery that keeps a model correct has to be live, forever—and the artifact you must debug in production is the one artifact you cannot read.

Four of the five are not independent problems. They are four symptoms of one architectural choice: the learning happens at runtime.

The fifth is the odd one out, and it’s worth saying why rather than rounding it into the pattern. Expertise scarcity is about people, not architecture. But it’s the one that tells you the diagnosis is right. Everything else in a systems codebase can be maintained by a good systems engineer; the learned component can’t. That’s not a staffing problem. It’s a foreign object in the codebase.

The founding paper already knew

Here is the detail that convinced me this isn’t hindsight, and it has been sitting in the most-cited paper in the field since 2018.

The learned-index authors measured their first attempt—a two-layer, thirty-two-neuron network—at about 80,000 nanoseconds per prediction in TensorFlow, against roughly 300 nanoseconds for the B-tree traversal it was meant to replace. They did not make the model smaller and call it a day. They built a framework that never runs the model at inference at all. It takes the trained network, extracts the weights, and generates C++. Model execution drops to 40–50 nanoseconds—four orders of magnitude—and the thing that ships is a data structure, not a model.

That is the entire argument of this post, executed in the methods section of the paper that started the field, and then not named as a principle.

It kept happening. Cache-replacement researchers trained an LSTM offline, studied what it had learned, and shipped a support vector machine with the same accuracy at orders of magnitude lower cost. A networking group converted deep policies into decision trees: 27× lower decision latency for under 2% degradation. To meet a microsecond budget on a NIC, another group distilled a neural congestion controller into decision trees, 500× faster. Google’s Borg heuristic came from evolutionary code search, and its authors say plainly why they chose that over deep reinforcement learning: the code solution “offers clear advantages in interpretability, debuggability, predictability, and ease of deployment—essential qualities for a mission-critical system.”

Five groups, five subfields, one move: learn offline, then throw the model away and keep what it taught you, in a form the machine can execute cheaply and a human can read. Nobody named it. It kept being reported as an implementation detail on the way to a benchmark number.

Move the learning earlier

What changed recently is that the “throw the model away” step no longer needs a researcher to do it by hand.

An agent that writes code can look at your traces, your workload, your profiles, and produce an ordinary heuristic: branches, thresholds, a couple of tuned constants, a comment explaining the reasoning. What Google’s Borg heuristic demonstrates is that the output of that process can be four lines of arithmetic and still be worth a fraction of a percent of a global fleet.

Now re-read the five barriers. Inference overhead: gone—the shipped artifact is arithmetic. Silent degradation: transformed, because the artifact is static and fails the way code fails. Worst-case behavior: analyzable, because you can read it. Maintenance: it’s code in your repo, reviewed by your process, reverted by a git command, and debugged the way you debug code. Expertise: your systems engineers can maintain it, because it’s the kind of thing they already maintain.

I should also say that I am not alone out here, and recent papers get to parts of this first. A Berkeley group argues that systems research is unusually suited to AI-driven discovery precisely because our verifiers are cheap—checking reduces to running the artifact against a workload—and forecasts that humans retreat to problem formulation. And a Google group—veterans of exactly this line of work—argues that AI for systems is “AGI-complete”—that scheduling and code optimization in their general form demand the kind of reasoning the frontier labs are chasing. I think both are right about the mechanism. Where I want to go further is the part none of them makes precise: which decisions move, when they were frozen, and what discipline governs the move.

So my claim isn’t “never learn at runtime.” It’s: the runtime should carry the smallest possible amount of learning, and we spent a decade assuming it should carry all of it.

Which raises the obvious question—if learning shouldn’t live at runtime, where exactly should it live, and what’s the vocabulary for saying so? That’s the next post.

References