AI Test Oracle
An AI test oracle uses a machine learning model — often another LLM — to judge whether a test's actual output is correct, especially for outputs too subjective or too varied for a simple equality check, like evaluating whether a generated summary is accurate or a chatbot's response is appropriately helpful.
This directly addresses the classic "oracle problem" for AI-generated output: there's often no single correct string a test can assert equality against, since a generated summary or response can be phrased many different valid ways — an AI test oracle instead evaluates output against a rubric or criteria (accuracy, relevance, safety) the way a human reviewer would.
LLM-as-a-judge is the most common concrete implementation of this idea — using a capable model, given clear evaluation criteria, to score another model's output — though it introduces its own known reliability concerns (the judge model can have its own biases or blind spots) that need to be accounted for and periodically validated against human judgment.
