PRODUCTION SCENARIO
A company compares two Amazon Bedrock models that summarize delivery incidents. Subject-matter experts rate the summaries, but the ratings disagree: some experts reward short summaries and others reward summaries that keep every operational detail. The company needs ratings it can compare across the two models.
Which solution will meet these requirements?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Define shared rating metrics for every reviewer to apply.
A human evaluation produces comparable results only when every reviewer rates against the same metrics, so factuality and brevity have to be defined before anyone scores. Dropping the reviewers who disagree removes the operational-detail viewpoint rather than deciding whether it counts. An automatic job discards the expert judgement the company wants. Token count ranks length, which is the thing in dispute.