Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta

A video is about sports for its first six seconds, then turns political for half a second. Catching that across a hundred million plus videos is one of two problems Aditya Gautam works on at Meta; the other is unoriginal content, trivial to make with AI tools and corrosive to attribution. Both sit on messy data: adversarial uploads, multilingual on screen text, drift, and no ground truth. Mismatch across modalities is a solved embedding problem; mismatch within one modality over time is not, so three agents share it. A perceiver splits the video where temporal change happens, not at a fixed frame rate, and emits clip level embeddings, tags, and OCR. A reviewer runs the temporal analysis over that JSON and folds in live comments and sentiment. A retriever indexes topics, embeddings, and entities into inverted, vector, and graph stores for pulling similar clips and authors at inference time. Every agent runs on a small specialized VLM, not a frontier model, since he does not care whether it can code. That means pretraining the vision encoder on in house images because user generated content looks nothing like clean web data, instruction tuning to a JSON schema, and a DPO loop where daily production samples pass through an in house LLM judge and a human queue tracing failures to the tool call or retrieval step. Distillation and quantization then produce a table of model sizes. Evaluation goes beyond precision and recall to per node latency, reasoning budgets, token cost, and judge drift. Three optimizations keep most videos out of the pipeline: compressing similar frames, caching verdicts on viral content, and pruning by metadata so trusted creators skip it. Speaker info: - https://www.linkedin.com/in/aditya-gautam-68233a30/ Timestamps: 0:00 - Two problems on short form video at 100 million plus scale 1:20 - Messy data: adversarial, multilingual, drifting, no ground truth 2:03 - Modality misalignment inside a single video 3:01 - Unoriginal content, and why one agent is not enough 4:22 - Reviewer, perceiver, retriever 5:31 - Temporal analysis, then indexing into inverted, vector, and graph stores 9:37 - The reviewer adds live user signals 10:45 - Pretraining and instruction tuning specialized VLMs 13:28 - DPO from production samples with humans in the queue 16:05 - Distillation and quantization to make it scalable 17:13 - Holistic evaluation: nodes, reasoning budgets, drift 19:28 - Spatial temporal reduction, caching, metadata pruning 20:35 - Takeaways