Your AI Agent Is Confidently Wrong About Production — Willem Pienaar, Cleric

Point an agent at a production alert and it comes back with a confident root cause. Often it's just the first error log it found. Willem Pienaar, co-founder and CTO of Cleric, explains why agents that are great at writing code struggle to debug production. Prod has no tests or linter to push back, it's sprawling, always changing and often impossible to reproduce, and LLMs are trained to be decisive when debugging needs doubt. He walks through the most common failure modes (mistaking a symptom for the cause, not knowing what's normal, lossy sub-agent summaries, anchoring on past incidents) and the grounding techniques that fix them. He also shows how tracking real outcomes after a fix lets you calibrate an agent's confidence so you can trust it. In this talk: • Why production debugging lacks the feedback loop coding has • Four failure modes: symptom vs cause, missing baselines, abstraction loss, anchoring • Grounding: parallel theories, baselines, dependency graphs, timing and evidence • Verifying fixes against real outcomes to calibrate confidence SPEAKER Willem Pienaar, Co-founder & CTO, Cleric LinkedIn: https://www.linkedin.com/in/willempienaar/ X: https://x.com/willpienaar GitHub: https://github.com/woop LINKS Cleric: https://cleric.ai/ CHAPTERS 0:00 Intro 0:13 Writing code isn't running it 0:38 The back-and-forth with agents 1:48 What we'll cover 2:12 Prod has no verification 2:57 Why prod is hard for agents 3:52 Models are trained to be confident 4:37 Common failure modes 5:02 Mistaking the symptom for the cause 5:32 Not knowing what's normal 5:47 The abstraction problem 6:11 Anchoring on past incidents 6:41 Techniques that work 7:11 Generate many theories 7:46 Compute a baseline 8:26 Use a dependency graph 9:06 Use time 9:35 Propagate the evidence 10:10 Results 10:55 The cost tradeoff 11:20 How do you know it worked? 11:50 Tracking outcomes after the fix 12:25 Calibrating confidence 13:39 Bringing prod signal into dev Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #SRE #AIAgents #AIEngineer