How to Set Up Local LLMs on Windows in 2026

Editorial review

AI tools, model releases, pricing, licenses, and platform terms can change quickly. Verify the official source before production or commercial use.

Quick recommendation

The best Windows local LLM setup for most builders is intentionally simple: use Ollama for the repeatable runtime, LM Studio when you want visual model browsing, Open WebUI when you want a browser-based chat workspace, Continue when you want editor-based coding help, and Qdrant only when you are ready to test document retrieval.

Do not start by installing every AI tool you can find. Start with one model that runs comfortably, write down how it performs on your hardware, then add one layer at a time. That habit prevents the most common local AI failure mode: a complicated stack where you cannot tell whether the model, runtime, prompt, retrieval layer, or hardware is the problem.

Recommended Windows stack

• Runtime: Ollama for CLI, local API, repeatable pulls, and automation.
• Desktop testing: LM Studio when you want a visual model browser and local chat UI.
• Chat workspace: Open WebUI after Ollama is already working.
• Coding assistant: Continue or another editor tool after you know which local model is usable.
• RAG/vector search: Qdrant only after basic chat quality and model speed are acceptable.

Step 1: choose a hardware lane

Your hardware decides the first model you should test. A CPU-only laptop should start with very small quantized models. A 12 GB VRAM GPU can test useful 7B to 14B-class quantized models. Higher VRAM gives you more room for larger models, higher context windows, and smoother multitasking, but it does not remove the need to test quality.

• Low-end / CPU-first: use small models and short prompts first.
• 8 GB VRAM: test compact 7B-class quantized models before trying larger models.
• 12 GB VRAM: test 7B and 14B-class quantized models for chat, coding snippets, and summaries.
• 16 GB+ VRAM: test larger context windows, heavier coding models, and local RAG experiments.

Step 2: install Ollama and verify it works

Install Ollama for Windows from the official download page, then open PowerShell and verify that the local service is running. Ollama normally exposes a local API on port 11434, which is what tools like Open WebUI and coding assistants use to connect.

ollama --version
ollama list
ollama run qwen3:8b

The exact model tag you choose should match your hardware. Start smaller than you think you need. A local model that answers quickly is more useful than a larger model that barely fits in memory and makes every test feel slow.

Step 3: test prompts before adding tools

Before installing Open WebUI, Qdrant, agents, or coding plugins, run a small repeatable test set. This gives you a baseline. Use the same five to ten prompts whenever you change model, quantization, context, or runtime.

ollama run qwen3:8b "Summarize this setup in five bullets."
ollama run qwen3:8b "Write a PowerShell command to list large files in Downloads."
ollama run qwen3:8b "Explain what RAG means in plain English."

Track model name, rough response speed, memory pressure, answer quality, and whether the model followed instructions. This simple log becomes more valuable than generic leaderboard scores because it reflects your machine and your tasks.

Step 4: add Open WebUI for a better chat workspace

Once Ollama is working, Open WebUI is the easiest next layer for a browser-based local chat workspace. The official quick start recommends Docker for most users. On Windows, that usually means Docker Desktop plus a persistent volume so chats and settings survive container restarts.

docker pull ghcr.io/open-webui/open-webui:main
docker run -d -p 3000:8080 -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

Open the UI at http://localhost:3000. If Open WebUI cannot see Ollama, check whether Ollama is running on the host, whether the container can reach the host service, and whether your firewall or Docker networking is blocking the connection.

Step 5: add coding only after the model is useful

Local coding assistants are most useful for private snippets, scripts, shell commands, regex, SQL, small refactors, and learning. They are usually weaker than frontier hosted models for large multi-file changes. Treat local coding as a useful private assistant, not as a guaranteed replacement for careful review.

Start by asking a local model to explain code, draft tests, or find simple bugs. Do not connect write-capable agents to important repositories until you understand how the model behaves and have a review workflow.

Step 6: add Qdrant only when you need RAG

Plain chat does not need a vector database. Add Qdrant when you have documents, chunks, embeddings, metadata, and test questions. Run a small local Qdrant container first, then connect it to your app or RAG tool after you have a clear retrieval workflow.

docker pull qdrant/qdrant
docker run -p 6333:6333 -p 6334:6334 -v qdrant_storage:/qdrant/storage qdrant/qdrant

Troubleshooting checklist

• Model is too slow: use a smaller model, shorter context, or lower quantization target.
• Out-of-memory errors: close other GPU-heavy apps and test a smaller model first.
• Open WebUI cannot connect: confirm Ollama is running and check Docker host networking.
• Answers are weak: test a different model family before blaming the runtime.
• RAG answers cite the wrong source: inspect chunks, metadata, retrieval results, and reranking before changing the chat model.

Sources