PRODUCTION SCENARIO
A prompt revision raises overall accuracy from 88% to 90%. On non-English requests accuracy falls from 82% to 61%. Those users remain in scope.
What should the release decision rest on?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: A per-language threshold that every in-scope slice must clear to ship
An aggregate can improve while a segment inside it collapses, so the release needs a per-language bar that blocks the ship until the non-English slice clears it. Weighting the aggregate by traffic share leaves it an aggregate and still hides the regression. Publishing per-language numbers without a gate records the problem and ships anyway. Deferring the threshold to the next revision exempts the one revision already measured as harmful to a group in scope.