The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev

BM25 stands for Best Match 25, and the number is not a version. A group of researchers ran a long series of scoring experiments decades ago, the twenty fifth one worked best, and the name simply stuck. Jo Kristian Bergum has spent more than twenty years on search and retrieval, and his claim is that this thirty year old lexical function is making a comeback without having changed at all. What changed is the user. A model already knows entities, companies, dates, postal codes and product identifiers, so it can write queries that are far longer and far more specific than anything a person would type, and it can fire off a dozen in a row. The old AOL query logs showed people searching in two or three words, and human query logs still look about the same today. An agent is a fundamentally more powerful user of a dumb tool. The sharpest evidence comes from a deep research benchmark of 830 riddle like questions over roughly one hundred thousand web documents. Stuff the answer bearing documents directly into the context window and accuracy is high, even for older models, which means reasoning was never the bottleneck. Hand the same model a search tool instead and accuracy drops, because now it depends on query formulation and on the retriever. Bergum compares a context window to a floppy disc, about 1.4 megabytes then and roughly 350,000 tokens now before quality degrades, so something still has to decide what goes in. He closes on a pattern he likes: dump retrieved documents into a file system workspace and let the model use grep and the other primitives it is already trained on. Speaker info: - https://x.com/jobergum - https://www.linkedin.com/in/jo-bergum - https://hornet.dev/ Timestamps: 0:00 - A thirty year old scoring function makes a comeback 2:19 - Best Match 25, and where the name came from 3:37 - The function did not change, the user did 4:57 - A benchmark of 830 riddles 6:01 - Context windows are floppy discs 7:05 - Reasoning is not the bottleneck 8:37 - How a model formulates queries 9:44 - Which BM25 do you mean? 12:06 - Retrieved documents as a file system 14:42 - Classical evaluation is dead 16:32 - Four claims to take away