Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora

Legora's search latency once went from a 100 millisecond P99 to twenty seconds, and the cause was a packing problem invisible in the schema. Legal work arrives as projects, a project gets worked hard for a while and then is never opened again, and when the team sharded document chunks across four thousand partitions the dead projects landed in the same partitions as the live ones. Every query pulled an enormous partition into memory, evicted the last, and thrashed the cache. Jacob Lauritzen walks the whole migration honestly, from one search cluster for everybody, then one per region once clients on three continents each demanded their processing stay home, then into the general purpose database the team already ran, which is where it broke. What worked was making the project the unit of storage, one namespace each, so an idle one rests in object storage costing nothing. Simon Eskildsen then explains why that shape suits regulated customers so well. If the namespace is the atom, every namespace can carry its own encryption key and its own bucket, which is what banks and large firms actually mean when they demand physical isolation and the ability to revoke a key and make their data unreadable. The best detail is an experiment: rather than build encryption into the SSD cache, they disabled the cache outright to see how bad it would be, found the performance acceptable, and left it off. The back half is a crash course in storage engineering: why round trips dominate when object storage answers in 200 milliseconds, why clustering into a tree beats navigating a graph, and why full text search at web scale costs more than vector search. Speaker info: - https://x.com/Sirupsen - https://www.linkedin.com/in/sirupsen/ - https://sirupsen.com - https://sirupsen.com/napkin Timestamps: 0:00 - What Legora does, and two kinds of legal search 2:32 - One search cluster, then one per region 4:00 - What enterprises mean by physical isolation 4:48 - Moving search into the database they already ran 5:38 - How hot and cold projects thrashed the cache 6:40 - A namespace per project, and what it fixed 7:56 - Writing straight to object storage 9:58 - Why a namespace is the unit of encryption 12:07 - Legal research, and ten billion vectors 14:25 - Puffing data through the memory hierarchy 15:30 - Why a tree beats a graph on object storage 17:45 - How full text search actually works