Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

The routing weights inside OpenAI's inference load balancer used to come out of a feedback loop. Engines reported signals, a controller smoothed them into a score, compared it to the fleet average, and nudged each weight up or down. A proportional controller, Lu Zhang notes, with real virtues: many signals folded into one decision, and constrained engines balanced themselves. It also produced behavior nobody could explain well. Ask why one engine got a higher weight and there was no clean answer. Tune one property and another moved. Worst was the oscillation: shift traffic off a hot engine, it cools, the controller reads cool as spare capacity and sends the traffic back, and the bouncing wrecks the KV cache locality routing was meant to protect. Qianru Lao walks through what replaced it: a control plane with a global view of every CPU cluster and GPU engine, and a data plane in each cluster that answers the one synchronous question, which engine serves this request, from a cached snapshot of routing weights. Signals still flow, into an optimizer rather than a loop. Its goal is to minimize expected end to end latency across all traffic, counting network distance and engine side queueing, under hard constraints that every request is routed and no engine exceeds capacity. Her example of why nearest is not enough: one region sends 120 requests a second at an engine that serves 100, while an engine two regions away sits at 40 of 80, so the farther engine wins once you count the wait. Zhang closes with the protections: outlier penalties, retry budgets that tighten as utilization climbs to prevent retry storms, and load shedding as last resort. Speaker info: - https://linkedin.com/in/qianru-lao - https://openai.com - https://www.linkedin.com/in/luzhang1/ Timestamps: 0:00 - From engine signal feedback loops to explicit policy 2:44 - What makes inference routing different 3:41 - Early days: weighted consistent hashing 4:23 - Weights from a proportional controller 6:57 - Oscillation that disrupts the cache 7:38 - Control plane, data plane, and a global view 9:18 - Three paths: request, signal, and routing weight 12:03 - Why not the nearest engine 13:28 - Inside the optimizer 15:38 - Penalties, retry budgets, and load shedding