PRODUCTION SCENARIO
A Generative AI Engineer's evaluation dataset stores only the questions users asked a benefits assistant. Each question has a different correct policy exception, and the team wants a scorer that checks whether the answer names it.
How should the team set up this evaluation?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Store expected facts for each record in the record's expectations
An evaluation dataset record holds inputs plus optional expectations, which are the ground truth labels, and expected_facts is the reserved key naming the facts that should appear in the answer. Retrieved context is input to the app rather than a label, so it cannot say which exception is correct. A Guidelines judge applies one rubric to every record and cannot carry a different correct exception per question, and RetrievalGroundedness only checks that an answer follows its context, so it passes an answer grounded in the wrong policy.