PRODUCTION SCENARIO
A Generative AI Engineer ships a release that changes the retriever, the system prompt and the model in one deployment. The team's evaluation dataset score drops from 0.82 to 0.61, and the engineer has to find which of the three changes caused it.
What should the engineer do?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Restore the prior release and re-apply the three changes one at a time, scoring each
A score change can only be attributed to a component if one component moves at a time, so the engineer restores the prior release and re-scores against the same evaluation dataset after each single change. A higher sample rate collects more failures without attributing any of them, re-shipping all three together may restore the score but leaves the cause unknown, and a tone judge measures a dimension that did not regress.