RAG: from question to evidence to answer

Ask a model how long your team keeps production database backups and it cannot know. The recorded reply, with no retrieval, says so and then lists industry ranges from 7 to 35 days, with grandfather-father-son rotations and compliance baselines. The fact lives in your handbook, not in the model's weights, and retrieval augmented generation (RAG) is the pattern that puts it in front of the model at question time.

The film follows that one question through a small, real RAG architecture. The handbook is ten short passages about backups: an overview, the retention policy, staging snapshots, restores, encryption, monitoring and so on. Each passage and the question were embedded once with `openai/text-embedding-3-small` on 11 September 2026, and every score on the rack is the recorded cosine similarity. The retention page ranks first at 0.63, ahead of the backup overview at 0.59 and the monitoring page at 0.58. Those three go into the prompt under a rule to cite a passage id after every claim and to say so when the answer is missing, and the reply from `google/gemini-3.8-flash` at temperature 0 is one sentence: 35 days, tagged [P2]. The tag is what makes the answer checkable.

Then the film breaks it twice. Delete the retention page and the next three passages fill the prompt, including one about staging snapshots kept for 7 days. The model does not borrow that number: it replies that the passages do not contain the answer. That is a retrieval failure with a correct generation step, and no amount of prompt work would fix it. The fix is in the index: the page was missing, or chunked so that the number and the word "retention" ended up in different pieces.

The stale page is the part people miss. Add an outdated 2023 page titled "How long are database backups kept?" and it outranks the real policy, 0.67 to 0.63, because similarity rewards wording that matches the question, not freshness or authority. The real page still reaches the prompt at rank 2, so the answer still says 35 days [P2], but it adds that the 2023 handbook says 14 days [D]. Nothing here is a hallucination: every claim is cited and every citation is accurate. The answer is still worse, because the evidence was.

The same experiment without the citation rule is in the explorer: the model gave the same two numbers with no ids, so nothing on the page tells a reader which claim came from which source. Citations do not stop bad evidence, but they make it visible, which is the precondition for fixing it.

The takeaway is to evaluate retrieval separately from answers. Keep a set of questions with known answer passages and measure whether the right passage reaches the prompt and what outranked it. A refusal can be a retrieval bug, and a fluent, cited answer can still carry a stale page. The fixes differ too: retrieval problems are solved with better chunking strategies, metadata filters for dates and owners, hybrid search that mixes keyword matching with embeddings, and a reranker that reads each candidate against the question. Generation problems are solved in the prompt and the model. Agentic RAG, where the model decides when and what to search, and graph RAG, which retrieves along links between entities, change how candidates are found, but the same two-stage evaluation applies.

The honest caveat is size: one question, eleven passages, one embedding model and one recording per prompt. Another embedding model will rank these passages differently, and a larger corpus makes near-duplicate pages like the stale one far more common, not less.