Tokens Should Have Jobs — Katelyn Lesse & Angela Jiang, Anthropic

Given the same fixed budget of roughly 600,000 tokens, an agent that did nothing but execute scored 76 on a bench of financial analysis tasks. An agent that spent part of that identical budget asking a second agent for advice scored 89. Same tokens, different jobs, better answer. Katelyn Lesse leads platform engineering at Anthropic and Angela Jiang leads platform product, and their talk goes after an assumption buried in how most teams tune agents: that every token is fungible, so the only lever worth pulling is a bigger budget. They propose giving tokens jobs instead. An advising strategy splits an executor from an adviser it can call out to mid task. A grading strategy defines what good looks like as a rubric up front, then provisions a grader that scores each attempt and sends the executor back to iterate. A dreaming strategy reads the executor's own transcripts, writes findings to memory, and hands them to the next run. The sharper result comes from rescoring that bench the way a working analyst would. A profit and loss statement that is 80 percent accurate is not 80 percent useful, it is useless, because you cannot make up a cost number. So anything short of a perfect run counts as a failure. Under that lens the execute baseline passes about 42 percent of the time, which means roughly three runs to land one clean answer, and about 1.8 million tokens of true cost. Advise and grade reach the same place for considerably less. Which strategy wins depends on what you are optimizing, because token efficiency and single run reliability point at different answers. Speaker info: - https://x.com/katelyn_lesse - https://www.linkedin.com/in/katelynlesse/ - https://x.com/angjiang - https://www.linkedin.com/in/angelajiang/ Timestamps: 0:00 - Are all tokens fungible? 1:34 - Giving tokens jobs other than executing 2:25 - Advising, grading, and dreaming 4:28 - The financial analysis bench 5:45 - Holding the budget constant 7:00 - Why 80 percent accurate is useless 8:42 - The true cost of a perfect answer 10:34 - Building strategies from primitives