LLM Evaluation
LLM evaluation is the process of measuring how well a large language model's outputs meet quality, accuracy, and safety expectations — using benchmark datasets, human review, or another model acting as a judge (LLM-as-a-judge). Unlike testing deterministic code, there's rarely one exact "correct" output to assert against, so evaluation measures degrees of correctness instead.
Testing a normal function means asserting that add(2, 2) equals 4 — there's exactly one right answer. Evaluating an LLM means asking whether a generated response is factually accurate, relevant, appropriately formatted, safe, and consistent with a defined persona or tone — none of which has a single correct string to compare against.
Evaluation methods generally fall into three buckets: reference-based (comparing output against a known-good "golden" answer using metrics like semantic similarity, not exact string match), rubric-based (a human or another LLM scores the output against defined criteria — helpfulness, accuracy, harmlessness), and behavioral (does the model refuse when it should, cite sources when it should, stay within its defined scope).
A solid evaluation harness runs against a fixed golden dataset on every model or prompt change, tracking scores over time — the LLM equivalent of a regression suite, just measuring quality on a continuous scale instead of a binary pass/fail.
