The Cheaper AI Model That Costs You More - Rayan Krishnan, Vals AI

Vals AI builds benchmarks that frontier labs cite in their model cards, and Rayan Krishnan explains why those single scores hide a jagged frontier. In finance tasks like tax and Excel LBO modeling, the best model changes by task, and a model that's cheaper on paper can cost more and score lower because it burns extra tokens. He shares lessons from building benchmarks (the right accuracy metric, reward hacking, multi-turn work), a case study of a team that spent $1.5M on tokens in a month before cutting spend about 90% with internal evals, and why a firm's own work history is its best eval. Chapters: 0:00 Intro 0:23 What model cards tell you 1:18 The jagged frontier 2:20 AI spend is up, ROI is unclear 3:40 The cheaper model that cost more 4:43 Finance benchmarks and the Pareto curve 6:36 Lessons from building benchmarks 8:41 Agents solve the test, not the problem 9:55 Evals are hard to run 10:59 Case study: $1.5M in tokens in a month 13:22 Where is your firm's history of work? 14:10 What evals unlock Rayan Krishnan https://www.vals.ai Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!