Spec-Driven Testing: Why AI Tests Should Never Read the Code

When AI writes the tests from the code, the tests learn the bug. Then they pass it. We measured it. Six small functions, one planted bug each, two Gemini models. Unit tests written from the code passed the bug in 22 of 60 suites. The same models, writing tests from the spec (only each function’s docstring and signature, no code), passed it in 0 of 60. A test needs its own source of truth. Not the code. The spec. Full study, method and results: https://www.testmuai.com/blog/ai-generated-tests-pass-broken-app/   Kane CLI: https://www.testmuai.com/kane-cli/  Install: npm install -g @testmuai/kane-cli Chapters  00:00 The problem, and the result  00:12 The bug the test copied  00:28 The experiment: 22 of 60  00:47 Written from the spec: 0 of 60  01:01 How Kane CLI starts from the spec Method  TestMu AI study, October 2026. Six small CommonJS functions, one realistic bug planted in each. Gemini 3.1 Pro Preview and Gemini 3.8 Flash at default settings, 5 runs per function per model, so 60 suites per condition. Jest 29.7. These were unit tests on small functions, not full applications. One more finding: the 22 suites that passed the bug still averaged a 97.3% Stryker 8.7 mutation score. A high mutation score didn’t catch a wrong expected result. The study prompted the Gemini models directly; it did not use Kane CLI. Kane CLI reads your spec, lists what must be true, and you approve it. Reasoning owns the path. Verification owns the verdict. #SpecDrivenTesting #AITesting #kanecli-follow