Inference Latency Testing
Inference latency testing measures how long a model takes to produce a response — end-to-end and broken down by stage, such as retrieval, generation, and post-processing — under realistic concurrent load, tracked at percentiles (p50, p95, p99) rather than just an average, since tail latency is what users actually experience as slowness.
An average latency number hides the exact thing that matters most for user experience: a system with a low average but a long tail — most requests fast, a meaningful fraction very slow — feels unreliable in a way the average completely fails to capture. This is why latency testing reports percentiles: p95 or p99 latency shows what the slowest real users actually experience, which is usually the number that determines whether a feature feels "fast enough" in practice.
For LLM-backed features specifically, latency needs to be broken down by stage — retrieval time, generation time (which itself scales with output length and can vary a lot token to token), and any post-processing — since a slow retrieval step and a slow generation step point to completely different fixes. Testing under realistic concurrent load matters too: a model that's fast for a single request can degrade sharply under the concurrency a real production launch brings, especially if it's sharing rate-limited infrastructure with other features.
