MLA-C02ML Model and Foundation Model (FM) DevelopmentMediumQuestion 3 of 30
An ML engineer built two Amazon Bedrock knowledge bases over the same policy documents, one with fixed-size chunking and one with semantic chunking. Before choosing one, the engineer must measure which knowledge base retrieves passages that contain all the facts needed to answer 500 test questions. Subject matter experts have written a ground truth answer for every test question. The measurement must exclude the effect of any generation model.
Which solution will meet these requirements?
Show answer and explanation
BRun a retrieve-only RAG evaluation job on each knowledge base and compare context coverage
A retrieve-only RAG evaluation job meets the requirements. Its report is based only on the data retrieved from the knowledge base, and its context coverage metric measures how much of the information in the ground truth the retrieved texts contain, which is why it requires the expert answers. A retrieve-and-generate job scores generated responses, so helpfulness mixes the generation model into the result. A model evaluation job on the generation model scores that model's answers without either knowledge base, so it cannot compare them. Retrieve latency shows which knowledge base is faster, not whether its passages contain the facts.