Function Calling Testing
Function calling testing checks that a model correctly translates a natural-language request into a structured function call — the right function name, correctly typed arguments matching a defined schema — since most agent and tool-use systems depend on this structured output being valid before anything downstream can run.
Function calling sits at the boundary between the model's free-form reasoning and the deterministic code that actually executes something, so it needs to be validated like an API contract: does the output parse as valid JSON, does it match the declared schema, are required fields present, and are enum-constrained fields actually restricted to allowed values rather than the model inventing a plausible-looking one.
Schema validation catches syntactic failures, but the harder bugs are semantic — the call is syntactically valid but wrong for the request, e.g. calling get_weather(city="Paris") when the user asked about Paris, Texas. Test sets need ambiguous and adversarial prompts specifically designed to surface this, not just clean, unambiguous ones.
Because models are non-deterministic, the same prompt can produce a valid call on one run and a malformed one on the next — testing this reliably means sampling each prompt multiple times and tracking a pass rate, not a single binary result.
