PRODUCTION SCENARIO
An extraction prompt was written against two vendor invoices and is correct on both. On invoices from a different region it returns fields that do not appear in the document.
What is the best next step?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Assemble a held-out set covering both regions, with a pass threshold
Evaluation has to measure defined criteria on representative inputs the prompt was not written against, which means both regions and a stated bar, so a fix for one region cannot silently break the other. A set drawn only from the failing region loses that regression coverage. Folding the failing invoices into the authoring examples turns them into development data and leaves nothing held out. Shadow traffic gives no fixed set, so two prompt versions never face the same inputs.