Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave

On an agentic request, somewhere between 80 and 90 percent of the input is identical to the request before it, and prefill is the most expensive thing an inference stack does. That fact, Sitanshu Gupta argues, is why a cached input token is priced so far below a fresh one, and it shapes nearly every choice in the platform he now runs at CoreWeave, four months in. It has to serve two consumption models without forking. Serverless, where you pay per token and never see hardware, with a provisioned throughput tier for customers who know their traffic profile and would rather not share a noisy neighbor. And dedicated, billed per GPU hour behind a private gateway, where the customer picks the engine, decides whether prefill and decode are disaggregated, and owns the deployment. Underneath sit four workload shapes fitted together across time like Tetris. Agentic and chat look alike on paper, long inputs and short outputs, but agentic turns arrive with no human reading in between, so the latency budget is tiny. Voice and video are streaming and latency bound. Batch customers hand over twelve hours of work and say process it whenever, which is why dedicated capacity serving real time traffic by day can be scheduled to drain a batch queue overnight. The router prioritizes KV cache locality first and least loaded second, across heterogeneous GPUs in different zones, and between chat turns the cache is offloaded to high bandwidth storage rather than evicted, so the next question does not pay for prefill again. The two biggest levers lately are four bit quantization and speculators trained asynchronously on a customer's own data for better acceptance lengths. Speaker info: - https://www.linkedin.com/in/sitanshugupta Timestamps: 0:00 - Four months in, leading inference at CoreWeave 1:22 - Serverless, dedicated, and provisioned throughput 3:10 - Four workload shapes and a game of Tetris 5:15 - The request flow, from gateway to GPU 7:17 - Why the router is KV cache aware 10:21 - Scheduling batch into idle overnight capacity 11:05 - Offloading KV cache between turns 12:31 - Quantization and custom trained speculators