The AI Memory Wall: Why LLMs & AI Accelerators Hit a Bottleneck

Learn more about the Memory Wall here → https://ibm.biz/~NjOWdJHJy More GPU compute doesn't always mean faster AI models. Kaoutar El Maghraoui explains the memory wall, arithmetic intensity, and why LLM inference can become memory-bound or compute-bound. Learn how memory bandwidth, batching, and the roofline model impact AI performance. 00:00 Why Faster Chips Don’t Always Make AI Workloads Faster 00:32 The Memory Wall: The Hidden Bottleneck in Modern Computing 01:00 Compute Growth vs Memory Bandwidth Explained 02:39 Arithmetic Intensity: The Metric That Predicts Performance 03:48 Understanding the Roofline Model for AI Optimization 05:27 Compute-Bound vs Memory-Bound Workloads Explained 08:25 LLM Decode vs Prefill: Why Performance Varies 09:48 How Batching Improves AI Inference Efficiency 10:35 Why Leading LLM Serving Systems Rely on Continuous Batching 11:22 Practical Steps to Diagnose and Optimize AI Performance AI news moves fast. Sign up for a monthly newsletter for AI updates from IBM → https://ibm.biz/~nnWVjebwV AI was used in the creation of the transcript and metadata for this video. #aimodels #aiaccelerator #ai #machinelearning --------------------------------------------------------------------------------------------------------- Find us on YouTube: 🔵 IBM Technology: https://www.youtube.com/@IBMTechnology 🔵 IBM: https://www.youtube.com/@IBM 🔵 IBM Developer: https://youtube.com/@IBMDeveloperAdvocates