Zero-Shot Evaluation
Zero-shot evaluation tests a model's ability to perform a task correctly with no task-specific examples in the prompt — just an instruction — measuring how well it generalizes from its underlying training rather than from in-context demonstrations, and serving as a baseline to compare against few-shot performance.
Zero-shot performance is a useful signal because it isolates what the model actually "knows" versus what a well-crafted set of examples is compensating for — a task that only works well with five carefully chosen examples in the prompt is more fragile in production than one that works reasonably well zero-shot, since real user inputs won't always resemble the curated examples.
In practice, teams run the same evaluation set both zero-shot and few-shot and compare the gap. A large gap tells you the task is example-dependent and the few-shot examples need to be treated as a first-class, carefully maintained asset rather than an afterthought bolted onto the prompt; a small gap suggests the instruction alone is doing most of the work and the prompt can likely be simplified.
Zero-shot scores are also the fairer number to compare across model versions or vendors, since few-shot results can be inflated by examples tuned specifically to one model's quirks.
