Token Cost Evaluation
Token cost evaluation measures how much a test suite or production feature actually costs to run against an LLM, based on the number of input and output tokens consumed per call, multiplied by the provider's per-token pricing. It turns "does this feature work" into "does this feature work at a cost that scales," a question functional tests never answer alone.
Unlike a traditional API call, an LLM call's cost isn't fixed — it scales with how much text goes in (the prompt, chat history, retrieved context) and how much comes back (the completion), and both vary per request. A test suite that passes every functional assertion can still be quietly expensive: a verbose system prompt, an unbounded chat history, or a chatty completion style each add real dollars per call, multiplied across every user and every retry.
In practice, token cost evaluation means logging token counts (and their dollar cost, since providers price input and output tokens differently) alongside every test run, not just success/failure. Teams set budgets per feature — "this endpoint should cost under $0.01 per call" — and fail a test or flag a regression when a prompt change or a longer retrieved-context window pushes a call over that line.
The common trap is optimizing for cost in isolation and quietly degrading quality — trimming context or switching to a cheaper model can reduce token spend while also reducing correctness, so cost evaluation has to run alongside accuracy checks, not instead of them.
