How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI

Your AI agent wrote a confident stock report saying 116% of analysts agree. Nothing in your checks caught it. This is the workshop for fixing that. Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm) builds a full evals loop on a Claude-powered financial analyst agent: trace it with Arize Phoenix, read the traces, write code evals and LLM-as-a-judge evals, test whether the judge can be trusted, prove a prompt fix with experiments, upgrade from Claude Haiku 4.5 to 5.5 with evidence, and gate it all in CI with npm test. Speaker: Laurie Voss, Head of Developer Relations, Arize AI X: https://x.com/seldo https://arize.com Phoenix: https://phoenix.arize.com 0:00 Intro & setup 2:47 "Looks good to me" isn't a test suite 4:57 Traces vs. evals: the mental model 8:30 Code evals, LLM judges & human review 11:57 Why agents are confidently wrong 16:32 Step 1: Tracing with Phoenix 36:36 Step 2: Reading your traces 43:21 The 116% analyst consensus bug 50:36 Step 3: Code evals 56:11 Step 4: LLM-as-a-judge 1:00:30 Why the correctness eval failed every report 1:10:41 Step 5: Writing your own judge 1:24:45 Step 6: Testing your judge 1:38:55 Step 7: Experiments 1:48:30 Upgrading to Claude Haiku 5.5 1:50:55 Evals in CI 1:55:25 Where to start on Monday Live from the Sheraton New York Times Square Hotel in New York City, AI Engineer New York 2026 is an AI conference for engineers and technical leaders building production AI across banking, hedge funds, fintech, insurance, and asset management. Event: AI Engineer NYC Dates: October 12–14, 2026 Venue: Sheraton New York Times Square Hotel Learn more about the event: https://ai.engineer/nyc/2026# Subscribe for more AI Engineer talks, livestreams, and event coverage: https://www.youtube.com/@aiDotEngineer