RAG — Retrieval-Augmented Generation

Golden rule: RAG quality is born in retrieval, not generation. The model answers only as well as the context it gets. When answers fail, inspect what retrieval returned first. Usually the model is not lying. It was fed the wrong evidence.

When you need this

RAG makes sense when the model must answer over data it does not know: internal wiki, product docs, contracts, support tickets, policies. And those data are too large or too live to paste whole into every prompt. Typical signals: “chatbot over our docs”, “search that answers in a sentence”, “support that knows our product”.

It exists because LLMs have a cutoff, never saw your private corpus, and invent fluent answers when they have nothing to stand on.

Long context (hundreds of thousands to a million tokens) moved the boundary. It did not erase it. A small stable corpus can often go in whole. A large, dirty, multi-tenant corpus still needs selection.

How it works

Two phases. Indexing happens ahead of time: split documents into chunks, embed them, store vectors, and keep a full-text index for keywords. Query time: find candidate chunks (vector + BM25), rerank by true relevance, put a small top set into the prompt with the question. The model answers from that context and cites sources.

Mental correction: RAG is not “add a vector database”. It is a search system whose answer writer is a model. Everything you know about search quality applies twice.

Build it step by step

1. Evals before pipeline. Write 30–50 real questions with gold answers and source locations. Without that set you cannot tell whether a chunking change helped.

2. Start stupid. BM25 + top hits in the prompt. No embeddings, no infra. Often 80 % of final quality, and a baseline you can beat. For tiny corpora also try “stuff everything” with prompt caching.

3. Chunk by structure, not character count. Headings, sections, paragraphs. A chunk should be one coherent idea (~500–1500 tokens) with metadata: document, section, age. Half a sentence without subject helps nobody.

4. Embeddings as complement, not replacement. Hybrid (vectors + BM25) beats either alone. Vectors catch paraphrase; full-text catches exact names, codes and numbers.

5. Add a reranker. Cheap retrieval returns top-50; a cross-encoder re-scores; prompt gets top-5. Often the best effort-to-gain step in the whole pipeline.

6. Teach “I don’t know”. If retrieval is empty or weak, the model should refuse instead of inventing. Answer only from provided context and cite sources. Citations are also your debugger.

7. Measure each half. Retrieval: is the right doc in top-k (recall@k)? Generation: given the right context, does the model answer correctly? If you mix the metrics, you fix the wrong half.

RAG vs long context

Long windows can replace RAG for small stable corpora, one-shot document analysis and whole-document synthesis. RAG still wins for large, frequently changing, multi-tenant data where you refuse to pay a million tokens per question and need citations from a specific source.

Practical rule: measure both baselines on the same eval set. Neither “1M context” marketing nor “enterprise vector DB” decides for you. Often the winner is hybrid: RAG selects candidates, long context holds them together for multi-hop questions.

Common mistakes

• Tuning the prompt while retrieval is broken → print retrieval for 10 failed questions first.
• Sentence or fixed-size chunking → respect document structure.
• Embeddings only → hybrid with full-text; vectors blur “v2” vs “v3” and product codes.
• No reranker → similarity rank is not relevance rank.
• No evals → every change is vibes over millions of chunks.
• RAG on a corpus that fits in context → paste the 50 pages; keep RAG for 5,000.
• Long context as an excuse to skip retrieval design → a bigger window makes messy data more expensive, not cleaner.

When not to use it

Database questions (“how many orders in May”) want SQL, not RAG. Creative tasks without ground truth do not need retrieval theater. Small stable corpora often prefer long context plus cache.

Sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• Introducing Contextual Retrieval (Anthropic)
• Effective context engineering for AI agents (Anthropic)
• AI Engineering (Chip Huyen)
• OpenAI: Retrieval guide

What to remember

RAG is tooling around the prompt. It does not fix tasks the model cannot do. It fixes tasks the model can do but has no data for. Build it like search: eval set first, dumb baseline second, hybrid + rerank third. Long context is alternative and partner, not magic replacement. When answers fail, read the retrieved chunks, not the model’s confidence.