PRODUCTION SCENARIO
A Generative AI Engineer runs a RAG agent that retrieves 50 candidates per query. Recall@50 is 0.94, yet the LLM often answers from the wrong chunk, and end-to-end latency is 6 seconds with almost all of it in generation.
Which retrieval change best improves answer quality here?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Add a reranker over the 50 candidates and pass on the top few
Recall@50 of 0.94 says the right chunk is already in the candidate set, so the problem is ordering, not coverage. Reranking boosts precision by achieving high recall with fewer candidates, and reranking 50 results takes under a second, negligible when generation dominates the budget. Cutting to 3 keeps the same bad ordering and discards recall. Doubling chunk size adds irrelevant text to each candidate. Re-embedding improves first-stage ranking at the cost of a full re-index, and a reranker buys the same precision without it.
Official Source: https://docs.databricks.com/aws/en/ai-search/retrieval-quality ↗