Stack recipe · Reviewed June 2026
Private Local Chatbot Stack
A practical stack for running private chat over local or self-hosted models with a browser chat UI and clear review habits.
Bill of materials
Model runner
Runs local models via a REST API on localhost:11434 — the standard local inference backend
GUI-first model runner with a built-in OpenAI-compatible server on any port
Chat interface
Self-hosted browser chat UI; connects to Ollama or any OpenAI-compatible endpoint
Document workspace + chat UI with built-in vector store — good for small teams
Recommended models
24B efficient model — strong for general chat and summarization on 16–24 GB VRAM
Fast small model for daily chat; fits comfortably on 8 GB VRAM at Q4_K_M
Compact 14B reasoning model from Microsoft — strong output-to-size ratio
Deploy with Docker Compose
docker-compose.yml — Ollama + Open WebUI
version: '3.8'
services:
ollama:
image: ollama/ollama:latest
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
# Remove deploy block if running CPU-only:
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
restart: unless-stopped
ports:
- "3000:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- open_webui_data:/app/backend/data
depends_on:
- ollama
volumes:
ollama_data:
open_webui_data:Start the stack
docker compose up -d
Pull your first model
docker exec -it ollama ollama pull qwen3:8b
Open the chat UI
Sign up on first load — Open WebUI creates a local admin account with no external connections.
open http://localhost:3000
Ran the commands? Measure what your setup actually does → — speed, load time, and whether anything is silently throttling it.
Best for
Developers, small teams, and privacy-conscious builders who want a local chat workspace before adding document retrieval or automation.
Core tools
- Ollama
- Open WebUI
- LM Studio
- AnythingLLM
Recommended models
- Mistral Small 3.1
- Gemma 3 27B
- Phi-4 Mini
- Qwen chat models sized to your hardware
Hardware notes
Start with a modern laptop or desktop for small quantized models. A GPU with 12GB to 24GB VRAM improves latency and room for larger models.
Setup steps
- Install one local model runner before adding extra tools.
- Pull a small model that fits comfortably in your RAM or VRAM.
- Add a chat UI such as Open WebUI or LM Studio for daily testing.
- Create a small prompt set for privacy, summarization, and coding-adjacent tasks.
- Review logs, retention, and access before inviting more users.
Trade-offs
Local chat improves control, but it does not guarantee better answers. Model quality, hardware limits, and logging choices still need review.
Alternatives
- Use hosted models for higher quality when privacy requirements allow it.
- Use AnythingLLM if document workspaces matter more than general chat.
- Use Jan for an open-source desktop-first workflow.
Related resources
Not sure if your PC has enough VRAM for this workflow?
Run the Local LLM Hardware Checker →FAQ
Does local chat mean no data ever leaves my machine?
Only if every component is local and configured that way. Check model providers, connected APIs, telemetry, logs, and sync settings.
Should I start with the biggest model?
No. Start with a model that runs reliably, then test larger candidates if latency and memory allow it.
Get practical stack updates
Join the OpenSourcesAI update list for new stack recipes, tool notes, and developer-first comparisons.
For builders
Sponsor a clearly labeled stack placement
Sponsor and partner placements are labeled and reviewed separately from editorial recommendations. For sponsorship options, advertise with us. For submissions or corrections, use the submit page.