Guide
What Is RAG?
RAG stands for retrieval-augmented generation. In plain terms: the AI looks things up in your own documents before it answers. It is the standard way to make a model useful on private material — and it works nothing like the thing most people assume it is.
Editorial review
AI tools, model releases, pricing, licenses, and platform terms can change quickly. Verify the official source before production or commercial use.
The short answer
A language model only knows what it absorbed during training. It has never seen your contracts, your notes, or last quarter’s report. RAG is the standard fix: before the model answers, the system searches your documents, finds the passages that look relevant, and puts them into the prompt alongside your question.
The model then answers having just read those passages — exactly as if you had pasted them in yourself. That is the entire mechanism. The word “retrieval” is the searching part, “augmented” means your question got extra material attached to it, and “generation” is the model writing the answer.
Nothing about the model changes. It does not learn, remember, or absorb your files. Ask it the same question with the database switched off and it will know nothing about them, because it never did. This is the point most explanations skip, and almost every later misunderstanding traces back to it.
What actually happens when you ask a question
Steps one to three happen once, when you set the system up. Steps four to seven happen every time you ask something.
- 1. Your documents are split into chunks. A few hundred words each, because whole documents are too big to be useful search results.
- 2. Each chunk is converted into a vector. A small embedding model turns text into a list of numbers representing its meaning, so that passages about similar things end up with similar numbers.
- 3. The vectors go into a database. A vector database such as Qdrant or Chroma stores them and can find the closest matches to any other vector very quickly.
- 4. You ask a question. It goes through the same embedding model and becomes a vector too.
- 5. The database returns the nearest chunks. Usually three to ten passages, ranked by closeness of meaning rather than matching keywords.
- 6. Those chunks are pasted into the prompt. Above your question, typically with an instruction like “answer using only the context below”.
- 7. The model answers. It is doing an ordinary reading-comprehension task on text it was handed a moment ago.
Step six is where the magic turns out to be plumbing. There is no special model mode and no training. The prompt just got longer, and the extra text happened to contain the answer.
RAG is not fine-tuning
When people say “I want to train the AI on my documents”, they nearly always want RAG. Fine-tuning is a different operation that solves a different problem, and reaching for it first is an expensive detour.
| RAG | Fine-tuning | |
|---|---|---|
| What changes | The prompt, at question time | The model’s weights, permanently |
| Adding a document | Index it — seconds | Retrain — hours of GPU time |
| Removing a document | Delete from the index | Very hard; often retrain from scratch |
| Can cite its source | Yes — it knows which chunks it used | No |
| Good for | Facts, documents, anything that changes | Style, tone, output format, domain behaviour |
The rule of thumb: fine-tuning changes how a model says things, RAG changes what it knows. If your goal is “it should answer questions about our documentation”, that is entirely a RAG problem. The fine-tuning guide covers the cases where the other column really is the right answer.
When you do not need it
RAG adds two components to your stack and a new class of failure. It is worth skipping when:
- You have a handful of documents. Pasting the relevant one into the prompt is simpler, more reliable, and free. Modern models handle long context well.
- The questions are general knowledge. If the answer is not in your material, retrieval contributes nothing but latency.
- Plain chat is not working yet. Get a model answering reliably first. Debugging retrieval on top of an unstable setup means guessing which of two layers is at fault.
Reach for it when the opposite is true:
- You have more material than fits in a prompt, and cannot predict which part holds the answer.
- The content changes and answers must follow it.
- You need citations — which document, which passage.
- The material is private, which is the usual reason to run all of this locally.
What it costs you
The honest tradeoff, because most introductions stop at the happy path.
- Retrieval becomes the bottleneck. Once RAG is running, answer quality depends mostly on whether the right passage was found — not on the language model. Most disappointing RAG systems are search problems wearing an AI costume.
- Bad chunking produces confident wrong answers. Split a table down the middle and the model gets half a table with no indication anything is missing. It will answer anyway.
- Two more moving parts. An embedding model and a vector database, each with its own configuration and its own ways of failing quietly.
- It does not raise the ceiling. Retrieval supplies facts; it does not improve reasoning. A 3B model with perfect retrieval is still a 3B model.
For builders
Built an AI tool or open-source project?
Submit it for review or sponsor a featured placement on OpenSourcesAI. For sponsorship options, advertise with us. For submissions or corrections, use the submit page.
Frequently asked questions
Does RAG train the model on my documents?
No, and this is the single most common misunderstanding. The model is never modified. Your documents are stored in a separate searchable database, and at the moment you ask a question the relevant passages are pasted into the prompt alongside it. The model reads them the way it reads anything else you type. Delete the database and the model knows nothing about your files, because it never did.
Is RAG the same as uploading a PDF to a chat app?
That is a small version of the same idea. When a chat app lets you attach a document, it is doing retrieval on your behalf — chunking the file, finding the parts that match your question, and putting them in the prompt. A full RAG setup is that mechanism made durable and scaled up: thousands of documents, indexed once, searchable across every conversation rather than one.
Do I always need a vector database?
No. For a few dozen documents, plain keyword search or simply pasting the relevant file into the prompt often works better and is far less machinery. Vector databases earn their place when you have enough content that you cannot know in advance which document holds the answer, or when questions are phrased differently from the source text.
Why does it give wrong answers when the information is in my documents?
Almost always because retrieval failed, not because the model did. If the right passage never reached the prompt, the model is answering from general knowledge and has no way to know it is missing something. Check what was actually retrieved before blaming the model — the fix is usually chunking, the embedding model, or adding a reranker, not a bigger LLM.
Can RAG run entirely on my own machine?
Yes. The embedding model, the vector database, and the language model all have local options — a small embedding model such as BGE or E5, a database such as Qdrant or Chroma, and any model you already run through Ollama. Nothing has to leave your machine, which is usually the reason for doing this locally in the first place.
What is an embedding model, and is it different from the main model?
Yes, different job and much smaller. An embedding model converts a piece of text into a list of numbers that represents its meaning, so that similar passages end up with similar numbers and can be found by proximity. It does not generate text. Most are a few hundred megabytes and run comfortably on a CPU, so adding one rarely changes your hardware requirements.
Does RAG make a small model smarter?
It makes a small model better informed, not better at reasoning. Retrieval supplies facts the model did not have; it does not improve the model’s ability to follow complex instructions or reason through a multi-step problem. If answers are weak on general questions, the model is the layer to change.
Next step: build one
The local RAG stack guide walks through every layer with real commands — document preparation, embeddings, Qdrant, and testing whether retrieval is actually finding the right passages. If you are still weighing which vector database to start with, the comparison guide narrows it down first.