How to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada

Nathanael Weill leads a team building AI agents at Air Canada, where tasks are so specific there's no public benchmark and domain experts have little time to grade results. His fix is to write the answer first: start from a known answer, generate the input from it, and check whether the agent finds its way back. He shows how that took a past RAG project from one expert-graded test a week to dozens of automated tests a day, then applies it to a supplier-contract review agent with five ways to build a test set: past decisions, planted rule violations, removed clauses, changes that shouldn't matter, and contract templates with filled-in blanks. He also explains why he avoids LLM judges whenever code can check the answer. Chapters: - 00:00 Intro - 00:47 No benchmark, no experts - 01:41 A lesson from a telecom RAG project - 04:43 Weekly expert reviews were too slow - 05:38 Flip it: write the answer first - 06:33 From one test a week to dozens a day - 07:30 Supplier contract review - 10:08 How the contract agent works - 10:39 Five ways to build a test set - 12:31 Planting rule violations - 13:39 Removing clauses - 14:42 Changes that shouldn't matter - 15:38 Templates with blanks - 16:39 Avoid LLM judges when code can check - 18:07 Use expert feedback to build better tests Nathanael Weill, Air Canada: https://www.aircanada.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!