Homa: The End of TCP for AI Clusters — John Ousterhout, Stanford

John Ousterhout, Stanford Professor and author of A Philosophy of Software Design, turns our attention to the evolving nature of AI networking workloads and why traditional protocols like TCP and RDMA are becoming bottlenecks in modern data center environments! The Shift in Workloads Historical context: AI traffic was dominated by massive, long-running transfers (gigabytes of gradients), where throughput was the primary metric (2:19-2:42). Modern AI: Workloads, especially inference and agentic applications, now rely on frequent, small coordination messages (e.g., KV cache lookups, barrier synchronization). These small messages are highly sensitive to latency (3:09-4:02). The Bottleneck: When small synchronization messages are mixed with large traffic, they get trapped in queues (caused by incast), significantly increasing 99th percentile (tail) latency. This causes GPUs to sit idle, wasting expensive compute resources (4:24-5:34). Why Legacy Protocols Struggle Sender-Driven Congestion Control: TCP and RDMA rely on the sender to detect congestion, often via packet drops or delayed signals from switches. This process is inherently reactive and oscillates, leading to unstable performance (7:10-9:58). Byte Stream Model: These protocols view data as an opaque stream of bytes rather than discrete messages, making it difficult to prioritize short, critical tasks (10:05-11:05). The Homa Solution John introduces Homa, a clean-slate transport protocol designed for data centers (11:15-12:15): Message-Based: Unlike byte streams, Homa understands message boundaries, allowing it to predict traffic and prioritize short messages using Shortest Remaining Processing Time (SRPT) (12:22-13:28). Receiver-Driven: The receiver controls the flow by issuing grants to senders, effectively managing congestion before it occurs at the switch (13:30-14:58). Priority Queues: Homa leverages the multiple hardware queues already present in modern switches to bypass long, queued traffic with low-latency short messages (14:59-15:51). Performance Results: Benchmarks show Homa can reduce tail latency for short messages by over 10x compared to TCP, while simultaneously improving performance for large messages (15:52-17:27). Speaker info: - https://x.com/johnousterhout - https://web.stanford.edu/~ouster/cgi-bin/home.php Timestamps: 0:00 - Why latency is becoming the metric that matters 2:19 - The old workload: gigabytes and throughput 3:09 - The new workload: metadata and coordination 4:24 - How one slow exchange stalls every GPU 6:07 - Incast, and where the queue actually builds 7:10 - Why congestion control lives on the wrong end 10:05 - A byte stream has no message boundaries 11:08 - Homa, and a clean slate redesign 12:12 - Messages, not streams 13:30 - Controlling congestion from the receiver 14:59 - Using the priority queues already in the switch 15:52 - The benchmark against TCP