Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse

A hands-on workshop on evaluating AI applications where mistakes are expensive. Annabell Schäfer and Doneyli De Jesus from Langfuse use a banking assistant to walk through the whole AI engineering loop. They cover tracing with PII masking, building datasets, turning domain-expert comments into failure modes, LLM-as-judge and code evaluators, and CI that blocks a prompt change when scores drop. Chapters: - 00:00 Why evals are hard - 05:01 The AI engineering loop - 12:47 Demo: a banking assistant with tracing and PII masking - 20:35 Building datasets - 25:18 Expert review and failure modes - 30:07 Choosing evaluators - 35:00 Running evals automatically on every prompt change - 39:44 Dashboards for stakeholders - 45:58 Working with your coding agent - 52:13 Q&A Doneyli De Jesus — https://langfuse.com Annabell Schäfer — https://langfuse.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!