AI-Powered Impact-Based Testing for Faster Safer Releases

In this session, 𝐒𝐫𝐢𝐤𝐚𝐧𝐭𝐚 𝐒𝐚𝐡𝐨𝐨, Staff Engineer at UKG, and 𝐍𝐚𝐯𝐧𝐞𝐞𝐭 𝐆𝐨𝐲𝐚𝐥, Principal Software Engineer at UKG, introduce the Smart Engine, which uses AI to identify the tests most relevant to each code change. Learn how it analyzes code modifications, dependencies, historical results, semantic relationships, and test reliability to predict impact - and runs only the subset that matters. Srikanta and Navneet also cover what that buys a team: shorter feedback time, no repeated cost of running the entire suite, and confidence that critical defects are still caught - with the engine learning from every cycle and adapting as the codebase evolves. 𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬: 0:06 Session Opens - AI-Powered Impact-Based Testing 1:07 Running the Whole Suite on Every Change Is Expensive and Slow 2:08 Speaker Intros - Srikanta Sahoo and Navneet Goyal, Principal Software Engineers at UKG 2:39 The Opening Question: What If Every Code Change Told You What to Run? 3:10 Ten Lines Changed, Thousands of Tests Queued 5:23 A Decade of Change: Smaller Commits, More Frequent Deployments 5:53 What Didn't Change: How We Run the Regression Suite 7:24 Test Execution Becoming the Delivery Bottleneck 8:56 The Right Question: Do All 50,000 Tests Have Equal Probability of Impact? 10:01 The Real Cost of Full Regression Is More Than Compute 10:32 The Cost Teams Don't Count: Developer Context Switching 12:07 The Regression Paradox: Two Bad Choices 13:10 Wanting Confidence That Can Be Explained and Defended 13:40 Test Selection Isn't New - Tags, Priorities and Severities 14:42 Why Each Breaks at Scale: Tags Need Discipline, Mappings Go Stale 15:13 The Missing Ingredient in All of Them: Understanding the Change 16:46 The Shift From Regression to Intelligent Regression 17:17 Identify Impact, Rank by Risk, Execute the Subset 18:20 The Types of Impact: Direct, Indirect, Semantic, Historical 19:21 Critical Path: A Small Change to Payments or Auth Deserves More 20:55 Combining 20+ Signals to Make the Selection Change-Aware 21:27 AI Recommends; Engineering Policy and Risk Controls Still Govern 21:58 Six Signal Categories: Semantic, Structural, Historical, Quality, Dynamic, ML 22:30 Structural: Import Analysis, Call Graph, Package Coupling 24:03 Semantic Signal: Comparing Business Meaning, Not Just Location 24:34 The Bank Analogy - River Bank vs the Bank With Your Money 25:36 Historical Signal: What Code and Tests Change Together 26:06 Failure History and Separating Real Defects From Flakiness 27:10 Quality Signal: The Trust Score and When to Widen Selection 27:43 Explainability: Why Each Test Was Selected 28:13 Dynamic Signal: Classifying the Type of Change 28:44 Risk Score: Estimating the Blast Radius 29:49 The Two ML Models: SGD for Linear, LightGBM for Non-Linear 30:50 A Real Example: Changing Password Length Validation 31:53 One Test Scored: Semantic 0.92, Dependency 1.0, Historical 0.85 32:55 The Team-Defined Threshold Selecting Three of Four 33:55 It Augments Your CI/CD - It Doesn't Replace It 34:25 The Stack: FastAPI, Python, BGE-M3 Embeddings, LightGBM, pgvector 35:27 Continuous Learning From Every Change and Every Execution 36:58 Accuracy Over Time: ~85% in Week One, ~98% at Six Months 38:02 The Measured Benefits: 90-95% Test Reduction With Recall Maintained 38:33 Validated Against a 100,000+ Test Codebase 39:36 The Headline Isn't Fewer Tests - It's Faster Confidence 40:07 The Production Question: What If the AI Misses a Test? 40:38 The Principle: The Model Proposes, Policy Governs 41:09 Lessons From Early Evaluation: The Skipped-Build Problem 42:42 Auto-Detecting Newly Written Tests So They Always Run Once 44:16 The One Idea: Running the Right Tests With Explainable Reasoning 44:46 Q&A: Should We Still Run Every Test for Every Release? 45:50 The Recommendation: Every Nth Run Should Be a Full Run 46:51 Q&A: Should Every Iteration Be Deployed? 48:25 Q&A: Generating Enterprise Test Data 48:57 Q&A: When an Agent Bypasses an Important Business Control 50:29 Human Diligence Still Needed in the First Quarter Register for TestMuConf 2027: https://www.testmuai.com/testmuconf-2027/?utm_source=youtube&utm_medium=organic&utm_term=&utm_campaign=ai_powered_impact_based_testing #TestMuConf #TestMuAI #TestSelection #CICD #TestAutomation #AITesting #DevOps #QualityEngineering