I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. — Keiji Kanazawa, Microsoft

A coding agent wired unit tests to production inference, then ran them about 30 times. Keiji Kanazawa had spent three weeks letting coding agents refactor a personal research assistant. A bad digest sent him looking for a trace, and he discovered the "agent" was still a deterministic workflow. Skills had no SKILL.md, different architectural concepts were aliases for the same thing, and a daily reflection job was producing empty reports. He had approved plans without checking whether the implementation meant what its labels promised. He rebuilt around an explicit agent runtime, clearer specifications and goal rubrics, comparing four frameworks before selecting tools for his own needs as a product manager. The working example is a video deep dive agent that extracts 440 keyframes from an inference lecture, pairs blackboard equations with explanations and lets him question individual sections. The result generalizes to slide decks, coding demos and terminal recordings. He iterates on an evaluation rubric, lets agents optimize against it and uses adversarial reviews between coding agents. The rubrics need scrutiny too: one was tuned to a single lecture and failed on five other videos. His answer to the code reading debate is to own the goals and evaluation loop while continuing to delegate most implementation. Speaker info: - https://x.com/gojira - https://linkedin.com/in/keijikanazawa - https://gojira.blog/ Timestamps: 0:00 - Introduction 0:33 - The L/Z debate over reading code 1:56 - Building a personal research agent 4:00 - The agent that was not an agent 6:33 - Unit tests on production inference 7:59 - Rebuilding with clearer specifications 9:21 - Comparing four agent frameworks 12:04 - Video deep dives from keyframes 13:59 - Rubrics and evaluation loops 16:36 - Catching an overfit rubric 18:25 - Own the goals and delegate the code