The Self-Improving OSS Agent Stack — Marc Klingen, Langfuse

Improving an agent used to mean a human reading traces every week. Now agents can do most of that loop for you. Marc Klingen, co-founder of Langfuse, shows how teams are moving from manually improving their agents to letting AI run the improvement loop. He explains the reference stack that emerged for building agents, which connects online tracing and monitoring with offline datasets, experiments and evals, and maps the layers of loops from next-token prediction up to human goal-setting. AI is already good at proposing fixes, and teams are starting to use it to maintain datasets and eval criteria too, while humans stay in charge of direction. In a demo, a coding agent uses Langfuse's agent skills to find that Langfuse's own changelog writer leaks internal jargon, updates the evals, and back-tests a better version before shipping it. In this talk: • The online + offline loop: traces, datasets, experiments and evals • Where humans still need to stay in the loop, and where AI can take over • Using implicit user signals like edits and approvals as feedback • Why owning your trace data matters when agents read far more of it SPEAKER Marc Klingen, Co-founder, Langfuse LinkedIn: https://www.linkedin.com/in/marcklingen/ X: https://x.com/marcklingen GitHub: https://github.com/marcklingen Website: https://marcklingen.com LINKS Langfuse: https://langfuse.com Langfuse on GitHub: https://github.com/langfuse/langfuse Langfuse on X: https://x.com/langfuse CHAPTERS 0:00 Intro 0:42 Loops, not prompts 1:12 How far models have come 1:42 The online + offline agent loop 3:06 From manual to automated 3:16 Layers of loops 4:06 Where humans still matter 5:01 AI proposing fixes 5:46 AI maintaining datasets and evals 7:00 The target keeps moving 8:10 Time invested vs quality 10:04 Feeding in user signals 10:24 Demo: a self-improving changelog writer 12:54 Demo: catching leaked jargon 13:53 Demo: back-testing V1 vs V2 15:13 The pattern the best teams use 15:58 Own your data layer Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #Langfuse #LLMEvals #AIEngineer