What is Retrieval-Augmented Generation (RAG)?
A technique that grounds a language model's answer in documents retrieved at query time instead of relying only on what the model memorized.
Definition
Retrieval-augmented generation (RAG) is a pattern where an application first retrieves relevant passages from an external knowledge source and then gives them to a language model along with the user's question, so the answer is based on that material. The term comes from a 2020 paper by Patrick Lewis and colleagues, which combined a pre-trained sequence-to-sequence model (parametric memory) with a dense vector index of Wikipedia (non-parametric memory) accessed through a neural retriever. Today a typical RAG pipeline splits documents into chunks, turns each chunk into an embedding, stores the embeddings in a vector database or search index, retrieves the most similar chunks for each query and inserts them into the prompt. Because the knowledge lives outside the model, it can be updated, filtered by permissions and cited without retraining the model.
When to Use Retrieval-Augmented Generation (RAG)
Answer quality is capped by retrieval quality: chunk size, the embedding model, how many chunks you retrieve (top-k) and whether you rerank all matter. Evaluate retrieval (did the right passages come back?) separately from generation (did the answer stay faithful to them?). Compared with fine-tuning, RAG is the usual choice when the knowledge changes often or must be cited; fine-tuning is better suited to changing style or behaviour.
Retrieval-Augmented Generation (RAG) in Practice
- A support chatbot embeds help-center articles, retrieves the five most relevant chunks for each question and answers only from them, linking to the source articles.
- An internal search assistant combines keyword search and vector similarity (hybrid retrieval) over wiki pages, reranks the results, then passes the top passages to the model.
- A coding assistant retrieves the files most related to a request from an index of the repository before proposing a change.
Common Mistakes
- Blaming the model when the retriever returned the wrong passages. Check what was retrieved before changing prompts or models.
- Chunks that are too large dilute relevance; chunks that are too small lose the surrounding context the answer needs.
- Assuming RAG eliminates hallucinations. The model can still ignore or misread the retrieved text, so ask for citations and check them.
- Skipping access control at retrieval time, which can leak documents a user isn't allowed to see through the generated answer.