PRODUCTION SCENARIO
A team ships prompt-template and tool-definition changes weekly. Two releases ago a minor prompt edit degraded compliance answers on edge cases; every unit test passed and the regression was found by a customer three weeks later.
Which gate catches that class of regression before the release reaches users?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: A fixed golden set rescored every build by schema checks and judge rubrics
A regression is a change against a baseline, so the gate must be a fixed set of cases scored the same way every build, pairing programmatic assertions with a judge calibrated on labelled examples. A canary is already in production, and rollback on error rate misses a well-formed but non-compliant answer. Online judging has no held-constant baseline and also fires after users are served. A schema validator re-checks the contract the failing releases already passed.