A language model only knows what was in its training text. It can't cite *your* private documents or this week's news, and when it doesn't know, it tends to make things up fluently. Retrieval-Augmented Generation (RAG) turns the closed-book exam into an open-book one: fetch the relevant pages first, then answer from them.
The two halves: retrieve, then generate
1Offline (index): split each document into chunks, turn every chunk into an embedding — a vector capturing its meaning — and store them in a vector store.
2Embed the question the same way, into a vector in the same space.
3Retrieve: find the chunks whose embeddings are nearest (highest cosine similarity) to the question's — these are the most *semantically* relevant passages.
4Augment: paste those chunks into the prompt as context.
5Generate: the model answers the question grounded in the supplied text, ideally citing it.
Search by meaning, not keywords
Embeddings place texts about similar things close together in vector space, so a question about 'refund policy' can retrieve a chunk that says 'returns are accepted within 30 days' even with zero shared words. That's the advantage over keyword search.
q_vec = embed(question)
chunks = vector_store.search(q_vec, k=4) # nearest neighbors by meaning
prompt = f"Context:\n{chunks}\n\nQuestion: {question}"
answer = llm(prompt) # grounded in retrieved text
Update knowledge without retraining
To teach a RAG system something new, you just add or edit documents in the store — no fine-tuning, no GPUs. This is why RAG is the standard way to give an assistant fresh, private, or domain-specific knowledge.
Garbage in, grounded garbage out
RAG reduces hallucination but doesn't eliminate it. If retrieval returns the wrong chunks — bad chunking, a poor embedding model, an ambiguous query — the model grounds its answer on irrelevant text. Retrieval quality is usually the bottleneck, not the LLM.
OperationTimeSpace
Index a document · embed + store onceO(chunks)O(chunks · dim)
Answer a query · k retrieved passages1 embed + ANN search + 1 generationO(k chunks)
Check yourself
Why can RAG answer questions about documents the model was never trained on?
