Pairwise Comparison Evaluation
Pairwise comparison evaluation shows a rater — human or an LLM judge — two candidate outputs for the same input and asks which one is better, rather than asking for an absolute quality score on a scale, since people (and models) are typically far more consistent at relative comparisons than at assigning stable absolute ratings.
Absolute scoring ("rate this response 1-10") suffers from calibration drift — the same rater's "7" can mean different things on different days, or different raters anchor to different baselines entirely. Pairwise comparison sidesteps this: "is A better than B" is a much more stable judgment to make consistently.
This is the basis of most model-vs-model leaderboards and A/B evaluation pipelines — running many pairwise comparisons between a candidate model and a baseline, then aggregating win rates (often via something like an Elo rating) into a single comparable score.
The main cost is combinatorics: comparing every pair among many candidates scales quadratically, so large-scale pairwise evaluation usually samples a subset of matchups rather than running every possible pair, and needs enough volume per matchup for the win rate to be statistically meaningful rather than noise.
