LLM-as-a-Judge
LLM-as-a-judge uses a capable language model to evaluate the quality of another model's output against defined criteria — accuracy, helpfulness, tone — providing a scalable alternative to human review for judging subjective or open-ended AI-generated responses.
Human evaluation of AI output is accurate but slow and expensive at scale — LLM-as-a-judge trades some accuracy for dramatically more throughput, letting a team evaluate thousands of model outputs against a rubric in the time it would take to manually review a few dozen.
It's not without known failure modes worth testing for directly: judge models can exhibit their own biases (favoring longer or more confident-sounding responses regardless of actual quality), and their reliability needs periodic validation against a smaller set of human-reviewed examples — the judge itself needs to be trusted, not assumed correct by default.
