How Netflix Evaluates AI When There's No Right Answer - Cameron Wolfe, Netflix

Cameron Wolfe, a staff research scientist at Netflix and author of the Deep (Learning) Focus newsletter, lays out a three-part framework for evaluating tasks with no single right answer, like creative writing. Human evaluation with calibrated guidelines is the North Star; LLM judges and rubrics, checked against human "golden" data, scale up offline evals; and A/B tests and production monitoring confirm results in the real world. Along the way he covers fighting noise with power analysis, error bars and resampling, and a Netflix project where an LLM judge's scores were causally linked to viewers abandoning shows. Chapters: 0:00 Intro 0:50 Why evals are hard: noise 2:43 The spectrum of verifiable tasks 3:54 A three-part framework 4:27 Human evaluation as the North Star 5:19 Calibrating human reviewers 6:05 How much data, and what kind 8:10 Offline evals: LLM judges and rubrics 9:47 Checking the judge against humans 11:45 Fighting noise with statistics 13:17 Production monitoring and A/B tests 15:14 Keep improving the loop 15:45 Recap Cameron Wolfe https://www.netflix.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!