Token-Level Evaluation
Token-level evaluation analyzes a language model's output at the granularity of individual tokens — the confidence or probability the model assigned to each generated token — rather than judging only the final assembled response, useful for detecting uncertainty or potential hallucination the surface text alone doesn't reveal.
Models generate text one token at a time, each with an associated probability reflecting the model's confidence in that specific choice — token-level evaluation looks at this underlying data, which is normally invisible in the final rendered text, to identify moments where the model was genuinely uncertain, even if the final output reads as fluent and confident.
Low-confidence tokens often cluster around exactly the claims most likely to be hallucinated or unreliable, making token-level analysis a useful complementary signal alongside output-level evaluation methods like LLM-as-a-judge — it's looking at a different, more granular layer of evidence about where a model's actual confidence was low.
