Model Provenance Testing
Model provenance testing verifies that the AI model actually running in production is the one that was tested and approved — the correct version, trained on the correct data, unmodified since evaluation — by checking model hashes, version metadata, and deployment logs, to catch silent substitutions like an untested fine-tune or a provider's undisclosed model update.
Unlike a traditional software deployment, where "what's running" is usually visible in a build artifact or image tag, an AI model behind a third-party API can change without a visible version bump — a provider updates the model backing an API endpoint, and every downstream test result and behavior guarantee is now describing a model that no longer exists.
For self-hosted or fine-tuned models, provenance testing means tracking a checksum or model ID alongside every evaluation result, so a test suite's "pass" result is tied to a specific model artifact, not just a model name — if that artifact changes, the test results are stale until re-run against the new one.
For third-party APIs where the underlying model isn't fully within your control, provenance testing shifts to monitoring: running a small, stable canary evaluation set on a schedule and alerting if the outputs suddenly shift in a way that suggests the provider changed something upstream, even though nothing in your own codebase did.
