version: '3.8'
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

  qdrant:
    image: qdrant/qdrant:latest
    container_name: qdrant
    restart: unless-stopped
    ports:
      - "6333:6333"
      - "6334:6334"
    volumes:
      - qdrant_storage:/qdrant/storage

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - QDRANT_URI=http://qdrant:6333
    volumes:
      - open_webui_data:/app/backend/data
    depends_on:
      - ollama
      - qdrant

volumes:
  ollama_data:
  qdrant_storage:
  open_webui_data:

Start all services

docker compose up -d

Pull chat model and embedding model

docker exec -it ollama ollama pull qwen3:8b && docker exec -it ollama ollama pull nomic-embed-text

Verify Qdrant is ready

curl http://localhost:6333/healthz

Open WebUI — configure RAG in Settings → Documents

Set Embedding Model to nomic-embed-text and Vector Database to Qdrant at http://qdrant:6333.

open http://localhost:3000

Ran the commands? Measure what your setup actually does → — speed, load time, and whether anything is silently throttling it.

Best for

Teams that need source-backed answers from private documents without sending data to a hosted app.

Core tools

Ollama
Qdrant
AnythingLLM

Recommended models

A local chat model that fits your hardware
An embedding model supported by your vector pipeline

Hardware notes

A modern desktop or small server can handle small corpora; add a GPU if you need faster indexing or lower-latency responses. Verify memory and storage against the model and document set you choose.

Setup steps

Start with one document source and one retrieval store instead of indexing everything at once.
Connect the local model runner and chat UI before tuning retrieval settings.
Index a small document set and check whether answers stay grounded in source text.
Adjust chunking, metadata, and refresh cadence before scaling to more files.
Review access, logs, and retention before sharing the stack with a wider team.

Trade-offs

Local RAG improves control, but answer quality still depends on document cleanup, chunking, embeddings, and hardware. Verify performance before production use.

Alternatives

Use Open WebUI Native Vector Pipeline if you prefer a simpler chat-centric interface.
Use a LangChain manual python script with Milvus if you want raw data engineering programmatic flexibility.

Related resources

Qdrant vs Chroma comparison Best local AI setup for beginners Local RAG stack guide

Not sure if your PC has enough VRAM for this workflow?

Run the Local LLM Hardware Checker →

FAQ

Can this stack handle larger document collections?

It can be used for larger document collections, but capacity and safety depend on file size, chunking, embedding throughput, storage, access controls, backups, and monitoring. Test with a small document set first, then scale gradually while reviewing permissions, logs, and retrieval quality.

Get practical stack updates

Join the OpenSourcesAI update list for new stack recipes, tool notes, and developer-first comparisons.

Get updates Browse guides

Private Document RAG Stack

Bill of materials

Full RAG stack via Docker Compose