500 Red Tests a Day: How Hugging Face Runs CI/CD for AI with Tarek Ziadé

What happens when your CI/CD pipeline can't give you a simple yes or no? Tarek Ziadé, who leads CI/CD on Hugging Face's Transformers library (and previously built CI at Mozilla), joins David Burns to talk about testing software where the maths changes every time you run it. Chapters: 00:00 Intro 00:35 Tarek's background: Mozilla to Hugging Face 02:57 The culture shock: non-determinism, GPU memory and 41-minute tests 08:28 How Transformers tests: CPU first, then two GPU lanes 12:25 13,000 daily tests and 0–500 reds 14:12 Triaging failures by trend, not by hand 16:08 Serge: the overnight agent that fixes tests (and reward hacks) 23:05 Lessons for any CI/CD: shrink the AI surface, cut the cost 29:08 What the Transformers library actually is 33:25 Backwards compatibility: the model linter and tiny models 40:39 Sandboxing agents: security and governance 45:33 Quickfire round 51:17 Wrap-up Follow Tarek's writing on the Hugging Face blog. Subscribe to BrowserStack Talks on Spotify, Apple Podcasts or YouTube so you don't miss future episodes. Get instant access to 3,500+ real mobile devices and browsers on our software testing platform that scales as your testing needs grow. Start for free - browserstack.com Join 7,000+ QA engineers on our Discord and discuss advanced testing strategies and stay ahead of industry trends: https://discord.com/invite/TGZmWQFg88 Follow us on ➡️ X - https://www.twitter.com/browserstack ➡️ LinkedIn - https://www.linkedin.com/company/browserstack/ ➡️ Facebook - https://www.facebook.com/BrowserStack/ ➡️ Instagram - https://www.instagram.com/browserstack © 2026 BrowserStack. All rights reserved.