How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs

Google went from 480 trillion to 3.2 quadrillion tokens a month. Here's everything the industry does to keep up. NEXT UP: AI ENGINEER CODE SUMMIT Join us in SF for the second annual AIE CODE Summit, Nov 10–12, our last flagship summit of 2026. Applications live now: https://aie.codes Daniel Kim of Cerebras and Michelle, co-founder of Gimlet Labs, explain how inference keeps getting faster and cheaper. Daniel covers the runtime: why decoding is memory-bound, splitting models across GPUs, continuous batching, KV cache, paged attention, prompt caching and chunked prefill. Michelle covers the system level: prefill/decode disaggregation, speculative decoding, attention/FFN disaggregation, Gimlet's multi-silicon cloud and an agent that writes optimized kernels. In this video: • Why decode is memory-bound and how big the KV cache gets • Paged KV cache, prompt caching and chunked prefill explained • Prefill/decode, speculative and attention/FFN disaggregation • Using agents to generate optimized kernels SPEAKERS Daniel Kim, Head of Growth, Cerebras X: https://x.com/learnwdaniel LinkedIn: https://www.linkedin.com/in/journeyer Michelle, Co-founder, Gimlet Labs LINKS Cerebras: https://www.cerebras.ai Gimlet Labs: https://gimletlabs.ai CHAPTERS 0:00 Intro 0:33 Exploding demand for tokens 2:02 Autoregressive decoding and the memory bottleneck 2:37 Splitting models across GPUs 3:27 Speed vs throughput Pareto frontier 4:27 Batching and continuous batching 5:22 KV cache memory growth 6:11 Paged KV cache 7:06 Prompt caching 7:41 Prefill vs decode 8:36 Chunked prefill for long contexts 9:38 System level: workloads differ 11:06 Prefill-decode disaggregation 12:21 Memory bandwidth: Cerebras vs H100 13:05 Cerebras and AWS Trainium 13:39 Gimlet's multi-silicon inference cloud 14:28 Speculative decoding 17:01 Attention-FFN disaggregation 18:25 Agents writing optimized kernels 20:16 Agents for silicon, compilers and ops Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMInference #AIInfrastructure #AIEngineer