Test Oracle
A test oracle is the mechanism used to decide whether a test passed or failed — a known correct output, a specification, a prior working version, or in some cases human judgment. Without a reliable oracle, a test can execute successfully but tell you nothing, because there's no trustworthy answer for what the "right" result should have been.
The simplest oracle is a hard-coded expected value: run the function, compare the output to a known-correct answer. But oracles get harder to define for complex or subjective outputs — is a generated image "correct"? Is a chatbot's response "good"? This is exactly the problem AI test oracles and LLM-as-a-judge approaches exist to address, using another model or a rubric to approximate a judgment a human would make, when a simple equality check isn't possible.
The "oracle problem" is a recognized, genuinely hard issue in testing: for some systems there's no efficient way to know the correct answer except running an independent, trusted implementation and comparing — which isn't always available. Where no oracle exists, testing shifts toward properties that should always hold (consistency, no crashes, bounds respected) rather than an exact expected output.
