Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta

Inference traffic at Meta already outpaces the largest microservices in the world, the fastest growing workload it has ever run. Nishant Gupta's frame for what comes next is 2008. The cloud began as virtual machines, then value climbed to schedulers, service meshes and autoscalers, until orchestration held the complexity and the value. AI is on the same path, compressed into a few years, with orchestration emerging around routing, KV cache management and prefill decode disaggregation. His sharpest point is coupling. None of the layers is new. What is new: a routing decision changes the cache hit rate, which changes batch composition, which changes GPU utilization, which changes the autoscaler's next move. A request is now a distributed transaction where every hop can retry, time out or fail; preempt a GPU after 200 tokens have streamed and you cannot simply retry. Reliability has to live in the control plane, the only layer that sees the whole workflow. A scheduler for inference has to see seven axes: GPU generation, memory headroom, KV cache state, whether weights are warm, tenant priority, latency budget, and workflow context, since failing step three of five wastes what steps one and two already spent. Every optimization fits one of four quadrants: avoid the work through caching, share it through batching, move it through routing to a cheaper model or region, or delay it through admission control. The metric to optimize is cost per successful task, not per token. Naman Ahuja takes over: observability as a control loop's input, the latency, cost and throughput triangle where every serving decision moves you somewhere, and the conclusion that inference needs its own control plane, the way virtual machines needed Kubernetes. Speaker info: - https://www.linkedin.com/in/nishantgupta-ai/ - https://buzzingtech.ai/ - https://www.linkedin.com/in/namanahuja/ Timestamps: 0:00 - Inference is the fastest growing workload Meta has seen 0:54 - 2008 again: value moves to the orchestration layer 3:13 - Microservices versus inference, dimension by dimension 6:09 - Old layers, new coupling 7:36 - A request is a distributed transaction 8:31 - Seven axes a scheduler has to see 10:19 - Avoid, share, move, or delay the work 12:36 - Cascading failure, with a KV cache twist 16:19 - The latency, cost, throughput triangle 17:00 - Inference needs its own control plane