The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

Byung-Gon Chun's team invented continuous batching, now standard across the industry, and the work that followed inspired one of the most widely used open source serving frameworks. So when he says agents have changed the economics of inference, he knows the tooling from the inside. His demonstration: the same coding agent, the same task of building a tower defense game, run once on a closed frontier model and once on an open weight model. Both finished at a usable level. The open weight run came in roughly five and a half times cheaper. That is the promise of open weights. But the model is only part of the bill, because agentic inference is a different problem from chat. In chat the unit was the request. In an agent the unit is the task: a loop of plan, act, observe, repeat, often running for minutes or hours, with sub agents fanning out in parallel and every observation appended to a context that only grows. His internal traces show consecutive steps sharing enormous prefixes, and recomputing that prefix on every call is compute spent on work already done. Nobody cares about the latency of one call; they care when the task is done. FriendliAI rebuilt its stack around that metric. Prefix caching so a shared prefix is computed once. A hierarchical KV cache across GPU, host memory and disk. Cache aware routing that sends a request to the replica already holding its prefix, instead of spreading load evenly and destroying locality. And agent aware scheduling that knows a call belongs to a longer program. One customer's split test found it seven times faster with a lower error rate. Speaker info: - https://www.linkedin.com/in/byung-gon-chun - https://bgchun.github.io Timestamps: 0:00 - The team that invented continuous batching 2:06 - One task, two models, one bill 3:18 - The unit is the task, not the request 5:25 - Consecutive steps share a huge prefix 7:02 - Four pillars of an agentic inference cloud 9:19 - Cache aware routing versus a naive load balancer 10:04 - Agent aware optimization 11:00 - The same task, end to end, on two providers