PRODUCTION SCENARIO
A Generative AI Engineer indexes 9,000 safety bulletins whose every page carries a 120-word legal disclaimer and a navigation menu. Retrieved chunks are often dominated by that repeated text, and the LLM answers with the disclaimer instead of the procedure.
Which change to the data pipeline addresses this?
Answering here is anonymous. Nothing is saved unless you sign in.
Show answer and explanation
Answer: Strip the disclaimer and menu text during parsing, before chunking
The RAG data pipeline calls for preprocessing extracted text to remove irrelevant or noisy information such as headers and footers, before chunking, so the boilerplate never reaches the embedding. Stripping it after retrieval cleans the prompt but leaves the vectors built from disclaimer-heavy text, so the right chunk still ranks badly and is often never retrieved to clean. Larger chunks carry the same disclaimer alongside more procedure, and near-duplicate removal misses boilerplate sitting inside otherwise distinct chunks.