Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI

A background agent quietly makes the tool call, then drops the result into the main model's context so the model believes it made the call itself. That sleight of hand is one of three tricks Bohan Li uses to run a voice agent on a slow, genuinely intelligent model without the caller noticing the wait. Li works on the voice harness at EliseAI and came from self driving, and he maps the cascaded voice stack onto that world: transcription is perception, the language model is planning, and speech synthesis is control. Each layer then gets optimized on its own terms. For transcription he runs two engines at once, a fast streaming one that emits immediately and a slower one with more context that can correct it, where a late correction is thrown away if newer audio has already arrived. The slower engine knows from the question being asked which part of the answer is a name and which is a date of birth. The synthesis trick is the neatest. A prefix cache watches the model's output stream and checks whether audio already exists for that run of words from an earlier turn. Openers repeat constantly in a scripted call, so the cache usually hits, and the agent starts speaking from it while the rest of the sentence is still being written. The synthesis provider is handed the whole sentence and generates it with natural prosody, unaware any of it has already played, and the overlapping audio is suppressed so the two halves join without a seam. Li closes by playing a real clinic booking call, where none of this is audible. Speaker info: - https://x.com/bobowchan - https://www.linkedin.com/in/bohan-li-7290b74a/ - https://eliseai.com/ Timestamps: 0:00 - Borrowing the self driving stack 1:52 - Perception: a speculative transcriber 4:24 - Eager generation and background tool calls 6:06 - Control: streaming speech synthesis 6:57 - The prefix cache 8:37 - Hiding the seam between cache and provider 10:21 - A full clinic booking call