Prompt Regression Testing
Prompt regression testing re-runs a fixed, versioned set of prompts against a model whenever the prompt, model version, or surrounding system changes, and compares the new outputs against known-good baselines — catching cases where a change that fixed one behavior silently broke another.
Prompts are code, in the sense that a small wording change can have a large, hard-to-predict effect on output — but unlike code, there's no compiler to catch a regression, and the same prompt can behave differently across model versions even with zero changes on the team's end. This makes prompt changes deceptively risky: a fix for one failing case can quietly degrade ten passing ones with no obvious signal.
The practical version of this is a versioned prompt test suite — a fixed set of representative inputs, run through the current prompt and model, with outputs diffed or scored against a recorded baseline. Because LLM output is non-deterministic, "diffed" usually means scored by a rubric or an LLM-as-a-judge rather than exact string matching, with a defined tolerance for acceptable variance.
Teams that skip this tend to discover regressions from user complaints instead of tests — every prompt change becomes a gamble on production traffic rather than something verified before shipping.
