Semantic Similarity Testing
Semantic similarity testing measures whether two pieces of text mean roughly the same thing, even if worded differently, using embedding-based comparison rather than exact string matching — essential for evaluating generative AI output, where a correct answer can be phrased in many valid ways.
Traditional assertions check for exact equality; semantic similarity instead converts both the expected and actual text into numerical embeddings and measures how close they are in that vector space, tolerant of paraphrasing, synonym use, and reordering that would fail a strict string match despite conveying the same meaning.
It's not a perfect substitute for human judgment or ground truth verification — two texts can be semantically similar while one is subtly, factually wrong in a way embedding distance alone won't catch, which is why semantic similarity is usually one signal among several (alongside ground truth checks and LLM-as-a-judge scoring) rather than the sole evaluation method.
