Shadow Deployment Testing
Shadow deployment testing runs a new model version in production alongside the live model, feeding it the same real traffic, but without serving its responses to users — comparing its outputs against the current model's to validate real-world behavior before it's ever actually exposed.
Pre-release evaluation sets, however thorough, are always a proxy for real traffic — shadow deployment closes that gap by testing the new model against the actual distribution of requests it will eventually serve, with zero user-facing risk, since its output is logged and compared but never returned to a real user. This catches the class of bug that only shows up with genuine production traffic patterns: unusual input formats, edge-case volumes, and real user phrasing that no test set fully anticipated.
The comparison itself needs to be more than "do the outputs match," since for generative models exact matches are rare and not even always desirable — teams instead compare distributions of key metrics (latency, refusal rate, length, a quality-scoring model's judgment) between the shadow and live model, looking for meaningful divergence rather than instance-by-instance equality. Shadow deployment adds real infrastructure cost, since every request now runs twice, so it's typically reserved for genuinely risky changes — a new base model, a significant prompt or fine-tune change — rather than every routine update.
