RAG Explained

RAG - retrieval-augmented generation - gives a language model access to facts it was never trained on. Instead of relying on memorised weights, the model receives relevant source passages at inference time and generates its answer grounded in them. The result is a system that stays accurate, cites its sources, and can be updated without retraining.

Why LLMs need retrieval

A language model's knowledge is frozen at training time. Ask it about an event after its cutoff date and it will confabulate - generating text that sounds plausible but is wrong. Even within the training window, long-tail facts that appeared rarely in the corpus are unreliable. Three structural problems follow:

• Staleness - model weights cannot be cheaply updated. A frontier model costs tens of millions of dollars to train and cannot be retrained weekly to reflect new documents.
• Hallucination - without a grounding source, the model must rely on parametric memory, which is compressed and lossy. It "fills in" missing facts with plausible-sounding fiction.
• Auditability - even when the model is correct, you cannot show the user where the answer came from. In regulated domains - legal, medical, finance - this is a blocker.

RAG solves all three. At query time you retrieve the relevant passages from an external corpus, inject them into the prompt as context, and ask the model to answer from them. The model's role shifts from "know everything" to "reason over supplied context" - a task it does well.

The RAG pipeline

RAG has two distinct halves: an offline ingestion pipeline that processes documents once (or whenever they change), and an online query pipeline that runs on every user request.

The ingestion path:

• Load - pull raw documents from PDFs, HTML pages, databases, or APIs. Format-aware parsers (PDFMiner, BeautifulSoup, Unstructured) extract clean text.
• Chunk - split documents into segments that fit the LLM's context budget and represent coherent ideas. Chunk size and overlap are the most influential tuning knobs in the pipeline.
• Embed - run each chunk through an embedding model to produce a dense vector that encodes its semantic meaning.
• Store - write the vector plus chunk text and metadata to a vector database. The database builds an ANN index for fast similarity search at query time.

The query path:

• Embed - embed the user's question with the same model used at ingestion time. Using a different model version breaks the metric space.
• Retrieve - search the vector database for the top-k most similar chunks by cosine similarity or dot product using approximate nearest-neighbour (ANN) search.
• Rerank - optionally re-score the top-k with a cross-encoder model for higher precision before passing to the LLM.
• Generate - inject the retrieved passages into the LLM prompt and generate the answer, instructing the model to cite only supplied context.

Chunking strategies

Chunking is the step that most influences retrieval quality. Too small and a chunk loses context; too large and it matches too broadly and wastes the context window. The right size depends on document type and query patterns.

Retrieval and hybrid search

Dense retrieval with ANN search is the backbone of RAG. The query vector is compared against all stored vectors using cosine similarity or inner product, and the top-k most similar chunks are returned. Modern vector databases handle this at millions-of-documents scale with sub-100 ms latency using ANN indexes like HNSW.

Dense retrieval has a blind spot: it captures semantic similarity but can miss exact keyword matches. A user searching for product number "SKU-8821" will get poor results from pure vector search because rare strings are poorly represented in embedding space.

Hybrid search fuses both signals. The most common approach is Reciprocal Rank Fusion (RRF): retrieve top-k from each system independently, then combine ranked lists using score = 1 / (k + rank) per result. Hybrid search consistently outperforms either approach alone across diverse query types and is now the production default at most AI teams.

Reranking

A bi-encoder retrieves fast but with imperfect precision - query and document vectors are compared independently, so subtle relevance signals are lost. A cross-encoder reranker receives the query and each candidate passage together, enabling full attention between them. This is slower (one model call per candidate) but significantly more precise.

Typical production setup: retrieve top-50 from the vector database, rerank with a cross-encoder (for example Cohere Rerank or a fine-tuned BERT), keep the top-5, and pass those to the LLM. The additional 100-300 ms latency pays for itself in answer quality, particularly for complex or nuanced questions.

Evaluation

RAG systems have two independent quality axes: retrieval quality (did we fetch the right chunks?) and generation quality (did the LLM use them correctly?). Evaluate them separately or you cannot diagnose which half is failing.

Build a small ground-truth dataset - 50 to 100 question/answer/source triples sampled from your actual document corpus - before shipping. Without it you are deploying blind and cannot tell if a change improved or hurt quality.

Failure modes

Most RAG failures fall into a small, predictable set of categories:

• Retrieval miss - the correct chunk is not retrieved. Root causes: wrong embedding model, poor chunking, query-document vocabulary mismatch. Fix: hybrid search, better chunking, or query expansion.
• Context stuffing - too many chunks dilute the signal. The LLM effectively ignores the relevant passage because it is buried in noise. Fix: rerank harder, reduce k to 3-5.
• Faithfulness drift - the LLM generates claims not supported by the context. Common with weaker models. Fix: add an explicit system-prompt instruction to cite only supplied text; upgrade the generation model.
• Index staleness - the vector index is not updated when documents change. Fix: event-driven or periodic re-ingestion pipeline with document-version tracking.
• Embedding model mismatch - query and document embeddings are produced by different model versions after an upgrade. Fix: always embed query and corpus with the same model, and version your vector index alongside your model.

For a deep dive into the storage layer, see Vector Databases. For how embedding models produce the vectors that power retrieval, see Embeddings Explained. To turn this knowledge into a role, read the RAG Engineer guide.