Building the Document Context Layer for AI Agents — Jerry Liu, LlamaIndex

A PDF is designed for printing, so its text is glyphs with coordinates and its tables are line segments with characters at cell positions. Reading order for a multicolumn page is not stored anywhere. That, Jerry Liu argues, is why an agent handed a raw file cannot make sense of it, and why document OCR is still unsolved after twenty years. His larger frame is that RAG in 2026 decomposes into an agent harness plus a context layer. Retrieval complexity has moved into the agent, which now reasons about the right search term instead of hacking around top k retrieval; context has moved up the stack toward MCP servers and skills; and programs are increasingly written in English. For LlamaIndex, what remains is the ten trillion plus pages locked in PDFs, PowerPoints, Word documents, and spreadsheets. The platform he describes has three layers: parsing into token efficient markdown and metadata, semantic storage as document management for humans and agents, and repeatable workflows for invoices, KYC, and claims. On parsing he contrasts heuristic pipelines with one shot VLM reads that hallucinate on text pages and cost a lot, and pitches a hybrid routing between cheap specialized models and frontier ones. ParseBench, his public benchmark of two thousand human verified pages, scores around fifty models on tables, charts, and faithfulness, and shows how much is left. Three regimes follow: high accuracy for regulated finance, low cost for indexing a million documents a day, and low latency for a thousand uploads in a minute, which is where LightParse comes in: a free Rust parser he calls the fastest open source option, the first pass before a VLM tool dives into a page. Speaker info: - https://x.com/jerryjliu0 - https://www.linkedin.com/in/jerry-liu-64390071/ Timestamps: 0:00 - RAG in 2026: an agent harness plus a context layer 1:33 - From naive top k retrieval to agents that reason about search 3:11 - Context moves up the stack, programs move toward English 5:15 - Context is everything: ten trillion pages in document containers 6:52 - Three layers: parsing, semantic storage, document workflows 8:27 - Why document OCR is hard: PDFs are drawn, not written 10:46 - Pipelines, one shot VLMs, and the hybrid in between 13:29 - ParseBench: 2,000 human verified pages, fifty models 15:08 - High accuracy, low cost, and low latency regimes 16:45 - LightParse in the agent loop 18:46 - Extraction with citations, document search, and what's next