PRODUCTION SCENARIO
In a failed RAG trace, the required paragraph appears in the retrieved context, but the generated answer states the opposite of what that paragraph says.
Which experiment isolates the failing stage?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Replay the generation step with the captured context held fixed
The trace shows the evidence already reached the generator, so retrieval is not the failing stage. Holding the captured context fixed and re-running generation varies only the prompt and the model, which is where the reversal has to come from. Re-embedding the corpus and raising k both change retrieval, which already produced the paragraph, and RetrievalRelevance scores whether the retrieved context matched the question, a check this trace has already passed.
Official Source: https://docs.databricks.com/aws/en/mlflow3/genai/tracing/ ↗