PRODUCTION SCENARIO
A team uses Claude to draft support replies and wants empathy graded on a 1-to-5 scale automatically. Their first grader prompt returns paragraphs of commentary the pipeline cannot parse.
How should the grader be set up?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Use a different model as grader, define the 1-5 scale, ask for the number
The guidance combines three things: grade with a different model than the one that generated the output, state the scale explicitly, and constrain the response to the number so it parses. Having the generating model grade its own reply is the arrangement the docs rule out. Asking for a rationale reproduces the unparseable prose the team already has. Pass or fail throws away the 1-to-5 resolution they asked for, so a reply that improves from 2 to 4 looks unchanged.