PRODUCTION SCENARIO
A company adds five examples to a prompt, and the model formats all five correctly. The company needs to estimate how the prompt will perform on requests it has not seen.
Which evaluation dataset will meet these requirements?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: A sample of real requests that the prompt was not built from
Only requests the prompt was not built from show whether the pattern generalizes, and a real sample carries the boundary cases that production will. Paraphrases of the five examples are still the distribution the prompt was tuned on. A public benchmark measures summarization in general, not this company's required format. Requests written by a second model reflect that model's habits rather than real users.