Non-Deterministic Output Testing
Non-deterministic output testing addresses the challenge that AI models can produce different, valid outputs for the exact same input on different runs — meaning traditional exact-match assertions don't work, and evaluation needs a different approach, like similarity scoring or a rubric-based judge, instead.
Traditional testing assumes determinism as a baseline: run the same input twice, get the same output, assert equality. Most generative AI systems break that assumption by design — asking the same question twice can yield two differently worded but equally correct answers, and a naive equality check would incorrectly fail the second one.
Practical approaches include: running the same test multiple times and checking for consistency within an acceptable range rather than exact equality, using semantic similarity instead of exact string matching, or using an LLM-as-a-judge to evaluate each output against criteria rather than against one single fixed expected string.
