PRODUCTION SCENARIO
An FAQ bot must give the same substantive answer when users phrase a question in different ways. The team wants an automated metric for this consistency.
Which evaluation method fits?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Embed each paraphrase's answer and compare them with cosine similarity
The docs describe cosine similarity of sentence embeddings as ideal for consistency, since similar questions should produce semantically similar answers even when the wording differs. Exact match penalizes harmless rewording, and ROUGE-L against the question measures the wrong thing.