Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google

Ask a benchmark harness for 200 queries a second and it may quietly deliver 38, then print results as though it ran 200. Ashok Chandrasekar opens with that experiment, which is why he and Jason Kramberger, both at Google, kept failing to reproduce published numbers. Python's global interpreter lock makes a single process harness CPU bound; the ones they tested capped near 170 and never said so. A thrashing client also inflates the latency it measures, once by 58 seconds, which reads as a bottlenecked server when the server was fine. A shared result claiming 20 percent better throughput turned out to have temperature set to zero, deterministic and faster than any real workload at 0.7. The same public dataset fed to two harnesses produced different input tokens, sampled and truncated differently. The diagnosis is usually your server. Often it is your harness. Kramberger presents the fix they built: Inference Perf, a CNCF project out of the Kubernetes serving working group. A main process schedules requests against a plan, Poisson, constant rate, or fixed concurrency, and fans them across worker processes that report when they actually fired versus when they were meant to. Client side telemetry sits beside server metrics, so you can tell a failing harness from a failing system under test. At 5,000 queries a second it kept up and said so. Configuration is declarative enough to replay multi turn conversations with length distributions, and a published workload catalog defines agentic generation, tree of thought and batch summarization in terms other tools can adopt. He closes with Prism, their UI under the llm-d project, showing combined optimizations against a plain Kubernetes service across eight replicas on TPUs. Speaker info: - https://www.linkedin.com/in/ashokchandrasekar/ - https://ashokc.dev - https://www.linkedin.com/in/jkramberger Timestamps: 0:00 - Two Google engineers who could not reproduce other people's numbers 2:47 - What a production scale benchmark has to do 4:34 - Four pitfalls: metrics, observability, reproducibility, data 5:29 - Ask for 200 QPS, get 38 6:35 - When the client inflates latency by 58 seconds 7:16 - Temperature zero and a 20 percent mirage 8:39 - Inference Perf: a multiprocess load generator 11:51 - A workload catalog others can share 14:02 - Principles for benchmark validity