Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story — Asaf Gardin & Yuval Belfer

Once in roughly a thousand prompts, the model returned gibberish. No crash, no warning, and high confidence, which made it an engineering problem, not a quality one. It happened only in vLLM, only under load, and only with Jamba, AI21's hybrid of attention and Mamba layers. Asaf Gardin and Yuval Belfer could not reproduce it with prompts, so they starved it: dropping vLLM's GPU memory utilization from ninety percent to twenty, at temperature zero, made request 8,854 return nonsense every time. From there they built a logprob comparison against a plain reference implementation, cleared the prefill kernel and a memory sanitizer, and saw the gibberish vanish when everything went through prefill. A false lead. The kernels were fine. They were being called at the wrong time for the wrong request. Nothing inside a forward pass has an identity, so they threaded the request ID down into a forward context, set a breakpoint on the guilty one, and watched the scheduler run decode before prefill. Attention survives that because it writes KV before it reads. Mamba reads its state first, so a fresh request computed over whatever came before. The second case looked like RL training instability: logprob spikes every twelfth step, before any weight update. Scaling rollouts per prompt from eight to 128 moved the spike to step one, and shrinking memory made it disappear, the wrong lever pulled. The cause was a 32 bit index that wrapped past four billion. The fix was one type, size_t. Two scenes, one criminal: both in the Mamba state cache, both surfaced by memory pressure, both found by logprob forensics. Stateful inference does not fail loudly. It lies with confidence. Speaker info: - https://www.linkedin.com/in/joseph-asaf-gardin/ - https://x.com/yuvalinthedeep - https://linkedin.com/in/yuval-belfer Timestamps: 0:00 - No crash, no warning, high confidence 2:48 - The one in a thousand gibberish 3:43 - Reproducing it fast by starving the GPU 5:18 - A logprob comparison against a plain baseline 8:48 - Threading a request ID through the forward pass 9:30 - Decode before prefill, and why only Mamba noticed 11:47 - Case two: logprob spikes every twelve steps 12:29 - A lever that changes the shape of failure 14:33 - A 32 bit index that wrapped around 15:42 - Two scenes, one criminal