PRODUCTION SCENARIO
A company must grade 15,000 generated responses for correctness and faithfulness before every release. Its subject-matter experts can review a few dozen responses at most. The company also needs a written rationale for every score.
Which approach will meet these requirements?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: A model evaluation job that uses an LLM as a judge
A judge-model evaluation uses a second LLM to score responses and explain how it scored each prompt and response pair, which is the rationale the company asked for. A work team of a few dozen experts cannot review 15,000 responses. An automatic job on built-in datasets scores AWS-supplied prompts rather than the company's own responses. A contextual grounding check filters responses at runtime and returns confidence scores, not a graded report.