PRODUCTION SCENARIO
An FAQ bot must give the same substantive answer when users phrase a question in different ways. The team wants a metric it can compute automatically, without a second model call.
Which evaluation method fits?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Embed each answer and compare the answers' embeddings pairwise
The docs name cosine similarity of sentence embeddings as the fit for consistency: paraphrased questions should produce semantically similar answers even when the wording differs, and values near 1 show agreement. Embedding the questions measures how close the paraphrases are to each other, which is a property of the test set. Comparing each answer to a stored reference measures accuracy, not agreement. Distance from a corpus centroid measures typicality, which a uniformly odd bot would pass.