Local RAG Stack with Ollama, Open WebUI and Qdrant

Editorial review

AI tools, model releases, pricing, licenses, and platform terms can change quickly. Verify the official source before production or commercial use.

What this stack is for

This stack is for private document question-answering, internal research search, local knowledge-base experiments, and early RAG prototypes. It is not the first thing every local AI user should build. Start with a working local model, then add retrieval only when you have documents and test questions.

Recommended architecture

• Ollama: local model runtime for chat models and compatible embeddings.
• Open WebUI: browser-based chat workspace and local model interface.
• Qdrant: vector database for chunks, embeddings, metadata, and retrieval experiments.
• Embedding model: use a model suited to your corpus and language needs.
• Evaluation set: a small list of questions with expected source documents.

Step 1: define the document job

RAG quality depends more on source preparation than on the chat UI. Before you install anything, write down which documents the system should answer from, who owns those documents, what should not be indexed, and what a correct answer must cite.

• Start with 10 to 50 documents, not your entire drive.
• Keep original filenames, paths, URLs, dates, and owners as metadata.
• Remove duplicates, drafts, sensitive files, and documents you do not have permission to index.
• Write 10 questions where you already know the correct source.

Step 2: run the base services

Get Ollama working first, then run Qdrant and Open WebUI as separate local services. This separation makes it easier to troubleshoot whether a failure came from the model runtime, the chat interface, or the retrieval layer.

ollama pull qwen3:8b
ollama run qwen3:8b

docker run -p 6333:6333 -p 6334:6334 -v qdrant_storage:/qdrant/storage qdrant/qdrant

docker run -d -p 3000:8080 -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

Step 3: choose chunking rules

Chunking decides what text the model can retrieve. If chunks are too large, answers become noisy. If chunks are too small, retrieval loses context. Start with consistent chunks, keep source metadata, then adjust only after testing retrieval results.

• Use stable document IDs so you can trace bad answers back to original files.
• Keep section titles and page numbers where possible.
• Do not mix unrelated files into one chunk just because they are short.
• Store enough metadata to filter by source type, owner, date, project, or folder.

Step 4: test retrieval before answer quality

When a RAG answer is wrong, the first question is not whether the chat model is smart enough. First inspect retrieval. Did the correct document appear? Was the chunk readable? Was metadata preserved? Did irrelevant chunks crowd out the answer?

Question: What does our refund policy say about annual contracts?
Expected source: policies/refunds-2026.md
Pass condition: correct source appears in top retrieved chunks before generation

Step 5: add a reranker only after baseline retrieval

A reranker can improve ordering after vector search, but it adds another model, more latency, and another place to debug. Add one only after you know your baseline retrieval is returning relevant candidates but ordering them poorly.

Security and privacy notes

• Do not index private folders automatically. Create an approved source list.
• Keep test documents separate from customer, employee, medical, legal, or financial records.
• Review logs before using real sensitive data. Local software can still store prompts, files, and outputs.
• Use read-only workflows until retrieval quality and permissions are understood.
• Back up Qdrant data only if you are comfortable backing up the indexed content and metadata.

Troubleshooting

• Answers sound confident but wrong: inspect retrieved chunks and add source-citation requirements.
• Correct document never appears: fix chunking, metadata, embedding choice, or query wording.
• Open WebUI cannot use the model: confirm the Ollama service is running and reachable.
• Qdrant data disappears: confirm that the Docker volume is mounted and not removed between tests.
• Everything is slow: use smaller models, fewer retrieved chunks, and shorter contexts first.

Sources