How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit

Jesse Hall of LiveKit shows how to build voice agents for financial services that feel human, not robotic. He explains why model leaderboards mislead (they score components, but you ship a conversation), why you should measure time to first audio and time to first sentence instead of first byte or first token, and how async tool calls keep a conversation flowing while slow back-end work runs. He also covers audio-aware turn detection, the roughly 1.5-second latency budget (500-600ms feels truly human), and benchmarking your own stack with simulated callers. Chapters 00:00 Intro: real-time agents vs background agents 01:47 Humanlike voice agents for financial services 03:46 LiveKit and the voice pipeline 04:21 Cascaded pipelines vs speech-to-speech 05:05 Why leaderboards don't pick your stack 06:46 Time to first audio, not first byte 07:52 Time to first sentence, not first token 08:38 Hiding slow tool calls like a human would 10:14 Demo: hotel booking agent 12:33 Turn detection: knowing when to listen 14:02 The latency budget 15:44 Benchmarking your stack with simulated callers 18:35 Results and a pre-launch checklist Jesse Hall, LiveKit Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!