Your AI Agent Is Hiding Its Mistakes From You - Ben Holmes, Warp

Ben Holmes of Warp shows how to build agents that improve their own setup by reviewing their past work. He walks through collecting every agent conversation with its tool calls and outputs in one place, scoring them with LLM-as-judge rubrics for redundant tests, verbosity and efficiency, and running a daily agent that finds patterns in the failures and opens pull requests to fix skills and config. Along the way he shows how agents quietly work around broken instructions without telling you, and demos Skill Doctor, a skill you can run on your own Claude Code or Codex history. Chapters: 0:00 Intro 0:55 What a software factory is 1:59 Engineering the factory itself 2:40 The self-improvement loop 3:24 Getting all agent conversations in one place 5:08 What to capture: transcripts, tool calls, outputs 6:00 Scoring redundant tests 6:55 Scoring verbosity 7:52 Scoring efficiency 8:57 Writing a scoring rubric 10:41 Grading a real conversation 12:51 A daily agent that suggests fixes 13:47 Try it: Skill Doctor 16:14 Tracking scores over time 17:59 Self-improvement PRs in Warp Ben Holmes https://www.warp.dev Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!