5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo

Nearly every intelligence gain in language models over the past year has come from letting them think longer. Voice agents have to turn thinking off, because the budget between a user finishing a sentence and the agent starting to speak is measured in hundreds of milliseconds. Venky B is founder and CEO of Plivo, which carries over a billion voice calls a month and has been building telephony infrastructure since 2011, and this talk is a tour of what breaks when a voice agent leaves the demo and meets production. On latency his numbers are blunt. Teams aim for under 550 milliseconds and most land between 750 and 1,200, and past that users simply hang up. His team's answer is smaller open source models hosted themselves, targeting under 300 milliseconds, chosen partly on how many tokens a language needs per word. The failure he says wrecks half of all deployments is data collection, and his fix is to stop treating it as transcription at all. Decide the shape before you ask. A phone number is a typed field with a length and a validator, so a stray letter in the middle is either corrected with confidence or sent back to the caller, and evaluation happens per field as a unit test rather than end to end. That reframing took his accuracy from roughly 30 percent to the mid nineties with no fine tuning. He is equally specific about transcription being brittle by default, especially with proper nouns and code switched languages, and about never feeding model output straight into speech synthesis. His own benchmark for a vendor is whether it can pronounce his surname and his company's name. Speaker info: - https://x.com/bevenky - https://www.linkedin.com/in/bevenky/ Timestamps: 0:00 - Where voice agents break between demo and production 2:20 - A billion calls a month 4:28 - The pipeline everyone builds first 5:34 - Failure one: latency and time to first audio 6:42 - Balancing cost, intelligence, and latency 7:48 - Why thinking models do not fit 9:56 - Choosing and sizing open source models 13:06 - Failure two: assume transcription is brittle 15:14 - Code switched languages break everything downstream 16:17 - Dynamic keyword boosting and LLM post processing 18:24 - Failure three: collect data as typed fields 20:31 - Relative dates and other traps 21:37 - Field level evals instead of end to end 22:47 - Failure four: normalize before synthesis 25:00 - Failure five: turn detection and barge in