Training Your Own Agent Models With RL Is Now Practical - Jason Mancuso, Modal

Jason Mancuso of Modal argues that treating models as swappable backends is a losing strategy, and that teams with agents in production already have what they need for reinforcement learning: a harness, evals and real traces. He explains why RL generalizes better than fine-tuning on recorded traces, then walks through a stable recipe for asynchronous RL built on GRPO with masked importance sampling, score centering and top-p sampling. He shows how sparse weight updates let Modal spread rollouts across clouds, GPU types and quantization levels while training stays stable and rollouts get 15% cheaper. Chapters: 0:00 Intro 0:20 Models aren't static backends 1:09 Why teams fine-tune 3:35 Why RL instead of SFT 4:46 You already have what you need 6:51 How reinforcement learning works 7:41 Async RL and staleness 9:38 A stable RL recipe 12:38 Why RL on Modal is different 14:55 Sparse weight updates 16:21 Mixing GPU types for rollouts 17:56 15% cheaper rollouts 19:16 Own your stack Jason Mancuso https://modal.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!