How We Built an Agent That Improves Itself — Zubin Aysola, Weights & Biases

Weights & Biases' ARIA agent now helps build itself. Zubin Aysola, Senior Software Engineer on Weave, shows live how ARIA takes a production trace, turns it into an offline eval task, finds the bug (a missing SDK call), writes a fix and benchmarks the new version against the one in production. The core challenge, Aysola says, is that benchmarks, evals and agent configs all change together, so measurement has to be airtight. His team keeps the production and research agents byte-for-byte identical. They define agent variants in YAML so they can run many side by side, give the agent an unconstrained sandbox, and score it in two ways: pass/fail and relative comparison. They run 886 tasks that include simulated multi-turn users. Every production miss, and every production win, becomes a new task. That flywheel lets the team spend its time on how to make the system better instead of writing benchmarks by hand. Speaker info: LinkedIn: https://www.linkedin.com/in/zubin-aysola Related links: Weights & Biases: https://wandb.ai W&B Weave: https://wandb.ai/site/weave Timestamps: 0:00 Intro: building ARIA 0:51 Why build your own agent harness 1:31 Benchmarks, evals and agents all change together 2:06 Weave for production and offline tracing 2:56 Live demo: ARIA doing autoresearch on itself 3:40 Turning a production trace into an eval task 4:50 Nightly CI evals 5:15 Building a simulation environment 5:50 Identical agents in production and research 6:40 Generate lots of traces 7:15 A model-agnostic harness and YAML variants 8:04 An unconstrained sandbox 8:54 The eval pipeline 10:14 Scoring: pass/fail and relative 10:54 Tasks as YAML: 886 tasks 12:03 Trajectories and the eval flywheel 12:53 Demo results: ARIA fixes its own bug 14:43 Staying in the loop 16:22 Wrap-up