PRODUCTION SCENARIO
A product manager writes the success criterion for a support-ticket classifier as "the model should classify tickets well." Engineering needs a criterion it can compute on every prompt change.
How should the criterion be rewritten?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: An F1 score of at least 0.85 on a held-out set of 2,000 labelled tickets
Good criteria are specific and measurable on the thing being judged; the docs contrast the vague "classify sentiments well" with an F1 target on a held-out test set, which a pipeline can rerun on every prompt change. A coverage rate measures well-formed output, and a classifier can label 95% of tickets validly and wrongly. A support lead's rating needs a person for each iteration. Latency is an operational metric a fast but inaccurate classifier passes.