Comparing two knowledge bases on retrieval alone

MLA-C02 ML Model and Foundation Model (FM) Development Medium Question 3 of 30

An ML engineer built two Amazon Bedrock knowledge bases over the same policy documents, one with fixed-size chunking and one with semantic chunking. Before choosing one, the engineer must measure which knowledge base retrieves passages that contain all the facts needed to answer 500 test questions. Subject matter experts have written a ground truth answer for every test question. The measurement must exclude the effect of any generation model.

Which solution will meet these requirements?

Show answer and explanation

BRun a retrieve-only RAG evaluation job on each knowledge base and compare context coverage

A retrieve-only RAG evaluation job meets the requirements. Its report is based only on the data retrieved from the knowledge base, and its context coverage metric measures how much of the information in the ground truth the retrieved texts contain, which is why it requires the expert answers. A retrieve-and-generate job scores generated responses, so helpfulness mixes the generation model into the result. A model evaluation job on the generation model scores that model's answers without either knowledge base, so it cannot compare them. Retrieve latency shows which knowledge base is faster, not whether its passages contain the facts.

Source: docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base…

Anonymous, no account needed. Your tally stays in this browser.

Next question Start 5 timed questions Browse all 30 original MLA-C02 practice questions