How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI

A hands-on workshop from SallyAnn DeLucia, head of product at Arize AI, on moving from vibe checks to real evals. She covers what an eval is and how it differs from a unit test, the main types (LLM as a judge, decision models, code evals, annotations, agent as a judge), how to read a trace, how to write concrete pass/fail criteria, and how to build and iterate on an LLM judge in Arize's prompt playground. Chapters 00:00 Intro and agenda 02:06 Why vibe checks don't scale 03:08 Is an eval just a unit test? 04:13 Why agents make evals harder 05:15 Types of evals 07:29 Workshop setup 10:16 Exploring the travel agent's traces 14:20 How to read a trace 19:03 Defining what good means 22:02 Writing pass/fail criteria 25:19 Building a dataset from traces 26:53 Building and testing an LLM judge 32:08 Next steps: alignment, online evals, Signal SallyAnn DeLucia, Arize AI Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!