RAG (Retrieval-Augmented Generation) fetches relevant documents from your own data and feeds them to an LLM at answer time — so the model responds from facts you control, not its training memory. It cuts hallucination and uses current, private data.
A large language model knows a lot, but two things it does not know: what happened after its training cutoff, and anything private to your business. Ask it about your return policy or last night's inventory and it will either refuse or, worse, make something up confidently.
RAG fixes this by separating knowledge from reasoning. The LLM stays the reasoning engine. Your data becomes an external library the model consults on demand.
The three steps
1. Index
Break your documents into chunks, turn each into an embedding, and store them in a vector database.
2. Retrieve
When a question comes in, embed it, and pull the handful of chunks closest in meaning.
3. Generate
Paste those chunks into the prompt alongside the question. The model answers using that supplied context — and can cite it.
Why operators reach for RAG first
Because knowledge changes and you need to audit it. Prices move, policies update, catalogues churn. With RAG you update the source documents and the answers update instantly — no retraining. And because the model is answering from retrieved text, you can show which source it used. That auditability is the difference between a demo and something you put in front of customers.
Where RAG breaks
RAG is only as good as its retrieval. If the right chunk is not retrieved, the model cannot use it — and may fall back to guessing. The hard engineering in RAG is not the LLM call; it is chunking, retrieval quality, and re-ranking. Get retrieval right and RAG feels like magic. Get it wrong and you have shipped a confident liar with a search bar.
RAG vs fine-tuning
They solve different problems. RAG changes what the model knows. Fine-tuning changes how the model behaves — tone, format, task shape. Most teams start with RAG because knowledge is what changes fastest, and layer fine-tuning only when behaviour needs to be locked in.