Large clusters for small models — Daniel Svonava, Superlinked

A single mid range GPU can turn half a million tokens per second into embeddings in the low tens of milliseconds, where a managed endpoint costs orders of magnitude more and takes hundreds. Daniel Svonava calls embeddings the no brainer entry point. His real subject is what comes after. A small model fits on one GPU two or three generations old, and for a specific task it is now at or beyond the frontier, which is flattening while small open models climb. You do not prompt one 27 billion parameter model for everything; you slice the workload into tasks and pick the model trained for each, so a contract review agent ends up running nine. The model that has seen the most Vietnamese receipts wins Vietnamese receipt OCR. The models exist; serving a wide fleet is the bottleneck. Three things break. Open source serving tools ship untuned, so adopting one is a research project. Top down routers, built to spread one big model across GPUs, choke on many small requests because their view of worker state is always stale; utilization stalls near 30 percent. And LoRAs and overnight fine tunes turn every deployment into a conversation between AI and infrastructure engineers. Superlinked's answer is open source under Apache 2.0 from control plane to GPU: a gateway annotates a request without fully parsing it and drops it into a shared queue, and workers pull and form their own batches. That inversion doubled cluster throughput. A Rust sidecar abstracts fifty adapters over three runtimes, an autoresearch loop ships every model already tuned, and one output was a LoRA that cost 80 cents and lifted retrieval on German legal text by 18 percent. Speaker info: - https://x.com/svonava Timestamps: 0:00 - Small open source models, do it yourself 2:30 - Small models are catching the frontier 3:42 - One task, one model: a nine model contract agent 6:37 - Serving tools are do it yourself research projects 7:31 - Why top down routing chokes on small requests 8:42 - LoRAs, fine tunes, and the conversation that kills velocity 11:12 - A gateway, a shared queue, and workers that pull 14:37 - Workers form their own batches, and throughput doubles 16:28 - Three runtimes and a Rust sidecar 19:29 - Half a million embedding tokens a second on one GPU 22:38 - Pack models on the same GPU 23:33 - Autoresearch that ships tuned configs 24:17 - An 80 cent LoRA