PRODUCTION SCENARIO
A Generative AI Engineer runs a RAG agent that retrieves 50 candidates per query. Recall@50 is 0.94, yet the LLM often answers from the wrong chunk, and end-to-end latency is 6 seconds with almost all of it in generation.
Which retrieval change best improves answer quality here?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Add a reranker over the 50 candidates and pass only the top few on
Reranking raises precision while keeping the recall of a large candidate set, and reranking 50 results takes under a second, which is negligible when latency is dominated by generation. Cutting num_results without reranking discards the recall the system already has.
Official Source: https://docs.databricks.com/aws/en/ai-search/retrieval-quality ↗