Human Evaluation (HumanEval)
Quick answer
Human evaluation is the practice of having people directly review and score AI-generated output for quality, correctness, or preference — the ground-truth standard that automated metrics (BLEU, ROUGE, exact match) are ultimately trying to approximate, used when nuance, taste, or real-world correctness can't be reduced to a formula.
Automated metrics exist largely because human evaluation is slow and expensive to run at scale — but for anything where "good" is subjective or context-dependent (tone, helpfulness, whether an explanation actually makes sense to a beginner), human judgment is still the most trustworthy signal available, and automated metrics are validated against it, not the other way around.
Reliable human evaluation needs its own quality controls: clear scoring rubrics rather than vague "rate 1-5" instructions, multiple raters per item to catch individual bias or inattention, and measuring inter-rater agreement — low agreement usually means the rubric itself is ambiguous, not that the raters are bad at their job.
In practice, teams often use human evaluation to build and periodically re-validate a smaller set of "golden" examples, then lean on cheaper LLM-as-a-judge scoring for day-to-day regression testing, spot-checking that judge against fresh human ratings periodically to confirm it hasn't drifted.
