Chain-of-Thought Evaluation
Chain-of-thought evaluation checks not just whether a model's final answer is correct, but whether its intermediate reasoning steps are logically valid and actually lead to that answer — catching cases where a model reaches the right conclusion through flawed reasoning, or shows sound-looking steps that don't actually support its final output.
A model can land on a correct final answer while its shown reasoning is wrong or irrelevant — sometimes called "right answer, wrong reasoning" — which matters because that same flawed reasoning pattern may produce a wrong final answer on a slightly different input where the errors don't happen to cancel out. Grading only the final answer misses this entirely.
Evaluating the reasoning chain itself usually means either human review against a rubric (does each step logically follow from the last, are there unjustified leaps) or an LLM-as-a-judge prompted specifically to check step-by-step validity rather than just final-answer correctness. Math and logic-heavy tasks are the clearest case, since each step is checkable in isolation.
It's also worth explicitly testing whether the shown reasoning is faithful to what actually produced the answer at all — models can generate a chain of thought that reads as the reasoning process but was actually constructed after the answer was already effectively determined, which overlaps with the same "post-hoc rationalization" risk explainability testing looks for.
