PRODUCTION SCENARIO
A prompt regression suite holds 480 stored cases, each with its own expected output. Nothing in the product waits on the run, it repeats on every model release, and each completion has to be scored against the case that produced it.
How should the run be submitted and reconciled?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: One Message Batch carrying a distinct custom_id on each of the 480 requests
Message Batches fit high-volume, independent, latency-tolerant work, and because results can return in any order, the custom_id is what ties a completion back to the case that produced it. Reconciling by line order scores cases against the wrong expected output, silently, whenever the order differs. One long conversation lets the cases contaminate each other and moves scoring into the model. Synchronous calls work but forfeit the batch discount and press the per-minute request limit for a run nothing waits on.