Choosing an evaluation set for a few-shot prompt

AIF-C01 Applications of Foundation Models Medium Question 27 of 65

A company adds five examples to a prompt, and the model formats all five correctly. The company needs to estimate how the prompt will perform on requests it has not seen.

Which evaluation dataset will meet these requirements?

Show answer and explanation

CA sample of real requests that the prompt was not built from

Only requests the prompt was not built from show whether the pattern generalizes, and a real sample carries the boundary cases that production will. Paraphrases of the five examples are still the distribution the prompt was tuned on. A public benchmark measures summarization in general, not this company's required format. Requests written by a second model reflect that model's habits rather than real users.

Source: docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html

Anonymous, no account needed. Your tally stays in this browser.

Next question Start 5 timed questions Browse all 65 original AIF-C01 practice questions