Agents That Self Improve While You Sleep - Fuad Ali, Arize AI

In this workshop, Fuad Ali of Arize AI shows how to build a self-improvement loop for AI agents: trace everything, let Signal (an always-on agent) read your traces on a schedule and file deduplicated issues, turn failures into evals and living regression datasets, run experiments to prove a fix works, and keep humans at the highest-leverage points: eval design, dataset curation and reviewing fixes. He demos the loop on a toy-shopping agent with a planted bug and takes questions on evaluating qualitative output, decision models and cost. Chapters 00:00 Intro and workshop setup 05:42 Where is your team on the maturity curve? 07:39 The loop: trace, signal, eval, experiment, fix 08:46 Demo agent and tracing 12:15 Signal: an always-on agent reading your traces 16:11 Agent Studio and managed agents 18:31 Turning failures into datasets 20:32 Evals: code, LLM as a judge, decision models 24:48 Creating an eval from a Signal issue 27:01 Experiments: proving the fix works 31:12 Keeping humans in the loop 34:16 Toward a fully automated loop 40:16 Q&A: evaluating qualitative output 48:21 Q&A: decision models, data sources and cost Fuad Ali, Arize AI Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!