Toxicity Testing
Toxicity testing checks whether a model's outputs contain harassment, hate speech, explicit content, or other harmful language — either because a user prompted for it directly or because the model produced it unprompted while generating otherwise-normal content. It's typically measured with both automated classifiers and human review.
Toxicity can surface in two very different ways: a user deliberately tries to provoke a harmful response (which overlaps with jailbreak and adversarial testing), or the model produces something harmful with no adversarial intent behind the prompt at all — an unprompted slur inside a generated story, or a biased generalization inside an otherwise helpful answer. The second case is harder to catch because it isn't triggered by an obviously risky prompt.
In practice, teams run a large batch of diverse prompts — some neutral, some deliberately edge-case — through an automated toxicity classifier (a smaller model trained specifically to score harmful content) to get broad coverage cheaply, then sample a subset for human review since classifiers themselves have known blind spots and false positives, especially around reclaimed language, satire, and non-English text.
A single toxicity score in isolation isn't very actionable — teams generally track it as a rate across a fixed, versioned prompt set over time, so a regression after a model or prompt change is visible immediately rather than discovered by a user report.
