Choosing an evaluation set for a few-shot prompt
AIF-C01
Applications of Foundation Models
Medium
Question 27 of 65
A company adds five examples to a prompt, and the model formats all five correctly. The company needs to estimate how the prompt will perform on requests it has not seen.
Which evaluation dataset will meet these requirements?
Show answer and explanation
CA sample of real requests that the prompt was not built from
Only requests the prompt was not built from show whether the pattern generalizes, and a real sample carries the boundary cases that production will. Paraphrases of the five examples are still the distribution the prompt was tuned on. A public benchmark measures summarization in general, not this company's required format. Requests written by a second model reflect that model's habits rather than real users.
Source: docs.aws.amazon.com/
Anonymous, no account needed. Your tally stays in this browser.