PRODUCTION SCENARIO
A Generative AI Engineer indexes 9,000 safety bulletins whose every page carries a 120-word legal disclaimer and a navigation menu. Retrieved chunks are often dominated by that repeated text, and the LLM answers with the disclaimer instead of the procedure.
Which change to the data pipeline addresses this?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Remove the disclaimer and menu text during parsing, before chunking
The RAG data pipeline calls for a cleaning step that strips irrelevant or noisy content such as headers, footers, and boilerplate before chunking and embedding, so it never enters the index. Larger chunks or prompt instructions leave the noise in the retrieved context.