Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face

Merve Noyan wrote a book on vision language models and now wants developers to stop calling them directly. Put one in front of a camera and you will never get real time; a small detector trained for the task runs at forty frames per second on a toaster and beats the VLM anyway. Her other complaint is licensing: people deploy a popular detector without noticing its copyleft license. So she built a toolkit that hands her favorite Apache 2.0 models to a coding agent, which she calls a clueless computer vision engineer, plus what she calls vibe training. Give it a dataset with no labels and it labels images with a nine billion parameter open VLM, passes the overlaid bounding boxes to two smaller VLM judges, merges their verdicts on minimum agreement rather than consensus, and trains RF-DETR. The whole run costs three or four dollars on Hugging Face jobs and inference providers. On road signs the trained detector lands a good mean average precision against ground truth, and on document parsing it generalizes, catching a signature the labeling model itself missed. The findings matter more: one judge rejects far more than the other, so consensus would have left too few examples; the judge prompts still need a human to approve them; and even the best coding agent flips traffic signs horizontally and jitters the color of traffic lights until told not to. She closes with the models as tools half of the toolkit, from a 600 million parameter model that segments the red car next to the orange car to pose, depth, and OCR picks, and plans for image guided detection where words cannot describe the part. Speaker info: - https://x.com/mervenoyann - https://www.linkedin.com/in/merve-noyan-28b1a113a - https://hf.co/merve Timestamps: 0:00 - Why developers should stop reaching for a VLM at runtime 1:51 - Read the license: move to Apache 2.0 models 2:46 - A toolkit for coding agents, the clueless computer vision engineer 3:41 - Vibe training: VLM as labeler, VLMs as judges, then train 5:40 - The pipeline: overlaid boxes, minimum agreement, RF-DETR 8:29 - What it costs: a few dollars end to end 9:40 - Results on road signs and document parsing 11:18 - Findings: judge imbalance, prompt approval, augmentation blunders 13:20 - Favorite models as tools, from segmentation to depth 15:52 - Future plans: image guided detection and IoU merging 17:57 - Q&A