PRODUCTION SCENARIO
An email from Anthropic announces that the model behind a company's invoice extractor is deprecated and gives a retirement date three months out. The extractor has a good evaluation set.
What should the team do first?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Run the eval set against the recommended replacement while there is runway
A deprecated model still works but requests fail after the retirement date, and the docs say to test replacements thoroughly well before that date, which an existing eval set makes cheap. Pinning the dated snapshot does not help, because retirement applies to the ID itself. Exporting the Usage CSV is the documented way to locate deprecated-model usage, but the scenario already names the one application affected, so it buys no information. And migrating first and monitoring afterwards moves regression discovery into production instead of into the eval.