Don't Approve the AI Model. Approve the App - Doneyli De Jesus, Langfuse

Doneyli De Jesus, who leads AI engineering solutions at Langfuse by ClickHouse, shares how his team helped a top-five bank put deployment gates on AI agents. Banks don't certify models, they certify applications, and they need setups that are on-premises, gated and air-gapped. He walks through building a golden data set, scoring traces across several dimensions, and how a small model that was 97% grounded but only 60% accurate on numbers cleared the bank's 85% gate once it got a calculator tool. Domain experts set the thresholds and review failures, taking evaluation from weeks to minutes. Chapters: 0:00 Intro 0:41 Banks don't care about public benchmarks 3:01 Certify applications, not models 3:18 On-prem, gated and air-gapped 5:00 Proving an app is ready 6:00 Golden data sets and traces 7:20 Grounded, but bad at math 8:52 Gating the whole workflow 10:34 Building the data set 11:32 Scoring: code, path and LLM judge 13:48 Setting gates with domain experts 15:53 Annotation queues and feedback 17:31 From weeks to minutes Doneyli De Jesus https://langfuse.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!