Few-Shot Prompt Testing
Few-shot prompt testing evaluates how a model's output quality changes when a prompt includes a small number of example input/output pairs before the real request, checking both that the examples actually improve performance and that the model doesn't overfit to quirks of the specific examples chosen.
Few-shot examples can help a lot — steering format, tone, and edge-case handling without fine-tuning — but they can also backfire in ways that are easy to miss: a model can pick up on an incidental pattern in the examples (like all of them being short, or all in a certain tense) and apply that pattern too rigidly to new inputs that don't share it, a failure mode sometimes called example leakage or spurious pattern matching.
Testing this well means varying the example set — swapping which examples are included, changing their order, and testing with zero, one, few, and many examples — to see how sensitive the output actually is to choices that feel arbitrary but measurably aren't. Prompt order sensitivity in particular is a well-documented LLM quirk worth explicitly checking for.
A practical habit: whenever a few-shot prompt is used in production, keep the exact example set under version control alongside the prompt itself, since silently editing "just one example" is exactly the kind of change prompt regression testing is meant to catch.
