In One Sentence

RAG (Retrieval-Augmented Generation) is a way to make a language model look things up before answering: the system finds the most relevant passages in a document collection, gives them to the model alongside the question, and the model answers based on that material.

Why RAG Exists

A language model’s knowledge comes from its training data, which causes three practical problems:

Knowledge has a cutoff. The model doesn’t know about anything that happened after training.

It doesn’t know your private data. Internal docs, product manuals and support tickets aren’t in public training data.

It can confidently make things up. When a model doesn’t know, it may still produce a plausible but wrong answer, the so-called hallucination.

RAG’s idea is simple. Instead of expecting the model to memorize everything, put the relevant material in front of it at answer time, like an open-book exam. You get fresh and private information in the answer, and the answer can point back to its sources.

The Basic Pipeline

A typical RAG system has an offline indexing stage and an online answering stage.

Offline (indexing):

  1. Collect documents: PDFs, web pages, wikis, database records.
  2. Chunk them: split long documents into smaller pieces, often a few hundred words each. Chunks that are too long mix in noise; chunks that are too short lose context.
  3. Embed them: an embedding model turns each chunk into a vector, so semantically similar text lands close together in vector space.
  4. Store the index: save vectors and original text in a vector database or a search engine with vector support.

Online (retrieve + generate):

  1. The user asks a question, and the same embedding model turns it into a vector.
  2. The index returns the chunks closest to that vector.
  3. The chunks and the question are combined into a prompt, such as “Answer using the material below. If the answer isn’t there, say you don’t know.”
  4. The model reads the material and writes an answer, often with citations.

What Actually Drives Quality

Retrieval sets RAG’s ceiling. If the right passage isn’t found, even a strong model will answer wrong. Common levers:

Chunking strategy. Split along natural structure like paragraphs and headings, and keep a small overlap between neighbors so sentences don’t get cut in half.

Hybrid search. Vector search understands meaning but can miss exact terms like product codes or names. Combining it with keyword search (such as BM25) is usually more robust.

Reranking. Retrieve a broad set of candidates first, then score them with a dedicated reranking model and keep the best few.

Query rewriting. User questions can be casual or missing context. Having a model rewrite them into search-friendly form often helps.

Metadata filters. Narrow by date, team or document type before searching. This is also a natural place to enforce access control.

RAG or Fine-Tuning?

They get compared a lot, but they solve different problems:

  • RAG adds knowledge. Good when material changes often, sources need to be cited or the corpus is large. Update the documents and the change takes effect with no retraining.
  • Fine-tuning changes behavior. Good for teaching a specific output format, tone or way of handling a task.

They also combine well: fine-tune the model to answer faithfully from provided material, and use RAG to supply that material.

Common Misconceptions

“RAG eliminates hallucination.” It reduces it noticeably but doesn’t remove it. Wrong or irrelevant retrieved passages, or a model that drifts from its sources, can still produce bad answers.

“With a big enough context window, you don’t need RAG.” Stuffing every document into the prompt is expensive and slow, and models get worse at finding key details in very long inputs. Retrieval is still an effective filter.

“RAG means a vector database.” Vector search is one way to retrieve. Keyword search, structured database queries or a web search API can all serve as the retrieval step.

“Set it up once and you’re done.” RAG needs ongoing evaluation. Build a set of representative questions with reference answers, then check separately whether retrieval found the right material and whether generation used it correctly. That tells you which stage to fix.

Further Reading