How Ramp Tests Its AI Bookkeeping Agent Before Customers Do - Ryan Stevens, Ramp

Ryan Stevens leads applied science at Ramp and explains how the team built end-to-end evals for Stack, its AI agent for bookkeepers. He starts by breaking an accountant's job into tasks like reconciliation and variance analysis, then builds synthetic businesses with full books so agents can be back-tested on the same data again and again. He covers accuracy versus pass@k, why the latest models still fail mostly on reasoning, and the experiments that didn't work, including auto-tuning prompts. Then he shows the ones that did: removing unneeded skills, testing memories across two time periods, and scheduled regression runs that report problems in Slack. Chapters: 0:00 Intro 0:47 What's your agent's goal? 1:16 Breaking down a bookkeeper's job 2:43 Building a benchmark 3:20 Before and after you have users 4:26 Data is the bottleneck 4:48 Building synthetic businesses 5:45 Evals are just back-tests 7:08 How each task is graded 9:14 Accuracy vs. pass@k 10:24 Picking a model 11:18 Why models fail: reasoning 12:18 Experiments that failed 13:36 Skill ablation 15:01 Testing memories 17:01 Regression testing in the background 18:07 Takeaways Ryan Stevens https://ramp.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!