21Retrieval-Augmented Generation

A model's weights are a snapshot: they freeze whatever the world looked like when pretraining stopped, and they hold nothing about your private wiki, last night's incident report, or the contract a user just pasted. Retrieval-augmented generation (RAG) is the standard fix. Instead of hoping the answer is baked into the parameters, you fetch the relevant text at query time and put it in the context window, so the model reads the facts rather than recalls them (Lewis et al., 2020). The whole chapter is one idea seen from several angles: for knowledge that is fresh, private, or must be cited, the context window is a better place to keep a fact than the weights.

21.1Why put facts in the context, not the weights

Three problems push you toward retrieval, and they are the ones an interviewer will name. First, the knowledge cutoff: a pretrained model cannot know anything that happened after its data was collected, and retraining to add a single document is absurdly expensive. Second, private data: your company's documents were never in Common Crawl, so no amount of scale puts them in the weights. Third, grounding: even for things the model does know, letting it answer from retrieved sources lets it cite them, and a model that must point at a passage hallucinates less than one answering from memory. The alternative to retrieval is fine-tuning the facts in, which is slow, has to be redone whenever the facts change, and still leaves the model unable to say where an answer came from. Retrieval decouples knowledge from the model: swap the corpus and the same weights answer new questions.

The distinction sharpens against the model's knowledge boundary (Chapter 10): the set of facts it actually holds. Inside the boundary the model can answer; outside it, an honest model should abstain, and retrieval is how you extend the boundary on demand rather than confabulating past it. Early work folded retrieval into pretraining itself — REALM learned a retriever jointly with a masked language model (Guu et al., 2020) — but the dominant 2026 pattern keeps the two separate: a frozen LLM plus a retrieval system you can update, index, and debug independently.

21.2Embeddings and vector search

How does the retriever find the right passage? Dense retrieval encodes the query and every document into vectors with an embedding model, then ranks documents by how close their vectors sit to the query's, usually by cosine similarity (Karpukhin et al., 2020). The premise is that a good encoder maps text with similar meaning to nearby points, so "how do I reset my password" lands near a support article that never uses the word "reset." This is where the encoder lineage from Chapter 4 pays off: embedding models are BERT-style bidirectional encoders, trained to compress a whole passage into one vector, precisely the job a causal decoder is not built for.

The catch is scale. Comparing a query against millions of document vectors one at a time is too slow, so production systems use approximate nearest neighbor (ANN) search: index structures like HNSW, a navigable graph of vectors, that find the near-best matches in logarithmic rather than linear time by accepting a tiny chance of missing the true closest one (Malkov & Yashunin, 2018). That approximation is almost always a good trade, because the ranking is already fuzzy.

21.3Chunking, reranking, and hybrid search

Naive RAG underperforms, and the reasons are unglamorous engineering rather than model quality. Documents are too long to embed whole, so you chunk them, and the chunk boundaries matter more than anyone expects: split too coarsely and one vector blurs several topics; split too finely and a chunk loses the context that made it meaningful. Practical systems use moderate chunks with some overlap so a fact straddling a boundary survives in at least one piece.

Two upgrades separate a demo from a system. The first is hybrid search: dense retrieval captures meaning but misses exact strings — product codes, error numbers, rare names — that a keyword method catches for free, so you run dense search and a lexical scorer like BM25, then merge the rankings (Robertson & Zaragoza, 2009). The second is a reranker: your first-stage retriever is fast but coarse, so it fetches a generous top-k (say 100), and a slower cross-encoder then reads each candidate together with the query and scores true relevance, keeping only the best few. The bi-encoder used for retrieval embeds query and document separately, which is what makes it indexable in advance; the cross-encoder cannot be pre-indexed but is far more accurate, so it is affordable only on a short list.

21.4Evaluating retrieval, and failing gracefully

RAG fails in two independent ways, and a good evaluation separates them. Retrieval can fail: the right passage never makes the top-k, measured with ranking metrics like recall@k and mean reciprocal rank against a labeled set of query-document pairs. Generation can fail even when retrieval succeeds, along two further axes: faithfulness (is the answer supported by the retrieved text, or did the model add unsupported claims?) and answer relevance (does it actually address the question?). Faithfulness is the axis RAG exists to improve, so it is the one to measure hardest — often with a second model judging whether each claim is entailed by the context (Chapter 24).

The most important behavior is what happens when retrieval returns nothing good. The failure mode to design against is a confident answer built on an irrelevant passage: the model faithfully grounds itself in the wrong text and is confidently wrong. When the top results are weak, the right move is to abstain — say the corpus does not cover this — not to confabulate. A system that knows when it has retrieved junk is worth more than one that always answers.

21.5Long context versus retrieval

If a model's context window now holds a million tokens, why not skip retrieval and paste in everything? Sometimes you should: when the relevant material is small enough to fit and changes every query, stuffing the context is simpler and avoids a retrieval pipeline's failure modes entirely. But three forces keep retrieval alive. Scale: a corpus of millions of documents will never fit a context window, and never will, because the corpus grows faster than the window. Cost and latency: attention makes every extra token of context more expensive to serve (Chapter 15), so re-reading a whole knowledge base per query is wasteful when a retriever could hand over the relevant page. Freshness: an index you can update beats a context you must reassemble by hand.

And long context has a quality problem of its own. Models attend unevenly across a long input: accuracy is highest when the needed fact sits near the beginning or the end and sags when it is buried in the middle — the "lost in the middle" effect (Liu et al., 2024). Simply dumping more text in does not guarantee the model uses it, which is another argument for retrieving a small, well-ranked set rather than a large, unordered one.

Retrieval is the harness component that decides what a model gets to read. The next chapter lets the model decide for itself — issuing its own searches and actions in a loop — which is what turns a retriever into an agent (Chapter 22).