Notes · Munna Suprathik
The RAG That Retrieved the Wrong Thing
By Munna Suprathik, Generative AI Engineer.
The demo went perfectly. That should have been the first warning.
A week later someone asked a slightly different question and my RAG answered with total confidence using a chunk that had nothing to do with it. The chunk was close in embedding space, so cosine similarity grabbed it. That is the trap nobody warns you about: an embedding squashes a whole passage into a few hundred numbers, and two passages that answer completely different questions can still land near each other. Bi-encoder search is fast because it compares those numbers, but fast and coarse are a package deal.
Close in vector space, far apart in meaning.
I fixed it in layers. First a cross-encoder reranker (Cohere Rerank) that actually reads the query and the chunk together and scores the pair. Retrieve fifty, rerank, keep the top five. Slower per query, far less wrong. Then HyDE, which sounds backwards until it clicks: ask the model to hallucinate a rough answer, embed that, and search with it, because a fake answer looks more like a real document than a short question does. Last, parent-child chunking. Embed small chunks so retrieval is precise, but return the parent section so the model gets enough context to reason instead of a sentence ripped out of its paragraph.
None of it was magic on its own. Stacked, they turned "close enough" into "correct."
Vector search finds similar, not right, and the whole game is closing that gap.