What should the team do before relying on the judge?

A judge model is an instrument, and an instrument that disagrees with the experts has to be checked against them: Google's method is to build an evaluation set carrying human ratings as ground truth, compare the model-based scores with those ratings, and inspect where they diverge. Rewriting the rubric to weight exceptions is the tempting move, and it guesses at the fix before measuring the gap, so the team cannot tell whether it worked. Adding the rejected answers enlarges a dataset scored by a measure that has not been validated, which produces more numbers of the same unknown quality. Running a second model's answers as a control varies the thing being judged rather than the judge, so a judge that ignores exceptions ignores them just as consistently.