Why Building an Eval Platform Is Harder Than It Looks — Braintrust

Most teams start their evals in a spreadsheet. Here's what happens when they try to grow out of it. Hossein Niazmandi, who leads solutions engineering in the West at Braintrust, explains why measuring agent quality is much more than putting a UI on a spreadsheet. He covers the two pillars of agent quality, evals before production and observability after, and why non-deterministic LLMs make both necessary. Then he walks through the stages teams go through: spreadsheets, a custom UI, side-by-side experiments for PMs and domain experts, and finally a flywheel that turns production failures into test cases. At that point the hard part becomes the data: traces that are huge, nested JSON, needing both real-time and long-running queries, which is why Braintrust built BTQL. He closes with agents running evals themselves and surfacing failures nobody thought to look for. In this talk: • Evals vs observability, and why you need both • The stages of eval maturity, from spreadsheets to a production flywheel • Why agent traces break typical data warehouses • Letting coding agents run evals while humans review the results SPEAKER Hossein Niazmandi, Solutions Engineering, Braintrust LinkedIn: https://www.linkedin.com/in/hossein-niazmandi-20a184bb/ LINKS Braintrust: https://www.braintrust.dev Braintrust SQL / BTQL reference: https://www.braintrust.dev/docs/reference/sql Braintrust on X: https://x.com/braintrust CHAPTERS 0:00 Intro 0:52 What Braintrust does 1:12 Evals and observability 2:21 Why evals matter 3:51 Isn't it just a spreadsheet? 5:11 What's under the iceberg 6:55 The improvement loop 7:35 Stage 1: spreadsheets 9:00 Stage 2: a custom UI 9:50 Stage 3: experiments for everyone 10:20 Stage 4: connecting production 11:49 Agent traces are huge and messy 13:19 Querying with BTQL 14:39 Agents running evals 15:53 Finding the unknown unknowns 16:43 Agents iterate, humans decide Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMEvals #AIAgents #AIEngineer