What's New in Inference Engineering — Philip Kiely, Baseten

TurboQuant reached twenty million people in March, and the memory stock index dipped because everyone assumed the KV cache had just halved. Philip Kiely had published Inference Engineering weeks earlier and watched a technique he had not covered go viral. So his team did the math. Four bit cache instead of eight does double effective bandwidth, but the extra decode computation cuts tokens per second by more than half: unacceptable in a data center, close to ideal on a memory starved machine in your basement. That split runs through the talk. Local inference is get it working, then make it less dumb. Data center inference is get it working, then make it less slow. And the biggest change since February is that data center optimizations increasingly come out of a dedicated training process, blurring the line between training and inference. He walks the big three. Quantization, where the data center answer stayed four bit weights rather than a compressed cache. Caching, where the question is now compaction: a learned bottleneck cross attends fixed query vectors against the full KV cache and emits compact keys and values in one forward pass. And speculation, where the field moved fastest. Small draft models from the same family were never good drafters; training one on the target's hidden states worked far better; then a diffusion drafter arrived that proposes eight or sixteen tokens at once, and in production it more than tripled acceptance over the previous best. Days before the talk, a newer method paired it with a sequential drafter. And retraining the speculator continuously on live prompts lifts acceptance twenty percent to double, if you can afford storage, compute, and permission. Speaker info: - https://x.com/philip_kiely - https://linkedin.com/in/philipkiely - https://baseten.co - https://philipkiely.com Timestamps: 0:00 - Three years at the World's Fair, and a book 2:30 - Two kinds of inference engineering: local and data center 3:52 - Training for inference blurs the handoff 5:20 - What happened to TurboQuant 7:37 - Where four bit quantization actually lands 9:28 - KV compaction and a learned cache 12:11 - Speculation, from small models to trained drafters 13:32 - DFlash: diffusion for drafting 15:11 - DSpark, and continuous speculator retraining 17:04 - What comes next