Skip to main content
bash TV

What Runs Where? Local AI in Production — Ahmad Osman, Osmantic

AI Engineer

2 views21 Nov 2026

YouTube

A capable local model still needs the right hardware, inference engine and tools. At CODE Summit 2026, Osmantic founder Ahmad Osman explains how to choose that stack for real workloads, covering quantization, memory bandwidth, batching and specialized models. He connects local AI performance to agent harnesses, evaluation traces and the economics of owning GPUs, then makes the case for continual learning on machines you control. Speaker info: - https://ahmadosman.com/ - https://github.com/theahmadosman - https://github.com/Osmantic/ODS Timestamps: 0:00 - What runs where: local AI in production 3:35 - Benchmarks, agent workloads and specialized models 11:00 - An agent-made trailer and the Portal harness 15:57 - Quantization and model quality 18:56 - Prefill, decode and choosing hardware 23:15 - Batching and inference engines per chip 27:57 - Pipeline optimization and speculative decoding 32:56 - Why local agents need search and a complete harness 36:13 - Choosing specialized models from real task traces 39:26 - Renting GPUs, costs and operational tradeoffs 44:20 - Planning for more capable models on existing hardware 48:07 - Why continual learning needs local AI 52:21 - Engrams, decision models and weight edits

Join the discussion

Have something to add?

Sign in to reply, follow this thread, and earn XP when your answer helps someone.