How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI

Xander Song, an engineer at Arize AI who works on Phoenix, introduces Harbor, an open source framework for evaluating agents on realistic tasks with real data in isolated sandboxes. He explains tasks, Docker environments, verifiers and rewards, trials and jobs, oracles that validate your setup before you spend tokens, and ATIF trajectories. Then he shows how his team runs Harbor at scale on Daytona and with Phoenix to test a prompt change that steered agents toward a new SQL tool, cutting tool calls without hurting correctness. Chapters 00:00 Intro: shipping on evidence, not vibes 01:28 What Harbor is 03:37 Architecture: tasks, environments, verifiers 06:08 Trajectories, jobs and oracles 07:58 Example: fixing a failing migration 10:48 Verifiers as black-box integration tests 14:21 Running at scale in parallel sandboxes 17:04 Harbor with Phoenix 20:37 How the Phoenix team uses Harbor 25:07 Q&A: how Harbor orchestrates runs 29:24 Q&A: complex systems and data size 33:21 Case study: steering agents to a SQL tool 38:34 Demo: datasets and experiments in Phoenix 44:10 Demo: comparing jobs task by task 48:31 Regression tests and skills Xander Song, Arize AI Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!