What Is a Local LLM?

Editorial review

AI tools, model releases, pricing, licenses, and platform terms can change quickly. Verify the official source before production or commercial use.

Plain-English definition

A large language model (LLM) is a neural network trained to predict and generate text. When you use ChatGPT, Claude, or Gemini, the model runs on the provider's servers and you interact with it over the internet.

A local LLM is the same kind of model, but instead of running on remote servers it runs on hardware you control: your laptop, desktop, workstation, or home server. The model weights — the billions of numerical parameters that encode what the model knows — are downloaded to your disk and loaded into your machine's memory when you run inference.

The practical consequence is that your prompts and responses never leave your machine. That is the central appeal for privacy-sensitive tasks, offline use, and cost control at scale.

How inference works locally

When you send a prompt to a local model, the following happens entirely on your hardware:

1. The runtime loads the model weights from disk into RAM or GPU memory (VRAM).
2. Your prompt is tokenised — converted into numerical tokens the model can process.
3. The model runs a forward pass through its layers to predict the next token, then the next, until it reaches a stopping point.
4. The generated tokens are decoded back into text and returned to your terminal, app, or chat interface.

Speed depends on how much of the model fits into GPU memory. When the full model fits in VRAM, inference is fast. When it spills into system RAM, it slows significantly. When it runs on CPU only, it is slower still but still usable for smaller models.

Key concepts you will encounter

What hardware do you actually need?

The right hardware depends on the models you want to run. Here is a practical starting guide by hardware tier:

• CPU-only (no GPU or integrated graphics): Run small quantized models (1B–3B parameters). Expect slow responses. Good for testing the workflow, not daily use.
• 8 GB VRAM (e.g. RTX 3070, RTX 4060): Run compact 7B-class quantized models comfortably. This is the most common starting point for practical local AI.
• 12 GB VRAM (e.g. RTX 3080 12G, RTX 4070): Run 7B and 14B-class quantized models. More room for context and multitasking.
• 16–24 GB VRAM (e.g. RTX 3090, RTX 4090): Run 14B–32B quantized models with large context windows. Capable enough for most local coding, RAG, and summarisation tasks.
• Apple Silicon (M-series unified memory): Unified memory architecture means RAM and GPU share the same pool. A MacBook Pro M3 Max with 48 GB or 64 GB can run surprisingly large models quickly.
• Multi-GPU or 48 GB+ VRAM: Enables 70B-class models at useful speeds. Primarily for researchers, production deployments, or dedicated AI workstations.

When in doubt, start smaller than you think you need. A 7B model running at 25 t/s is more useful than a 34B model running at 2 t/s.

Local LLM runners: which one to use

Most builders should start with Ollama. It is the simplest path from zero to a running local model with an API that other tools (Open WebUI, coding assistants, scripts) can connect to immediately.

Which open-weight model families to try

When local LLMs are the right choice

Privacy and data control

Privacy is the most common reason builders choose local models. When the model runs entirely on your hardware:

• Your prompts never leave your machine.
• Responses are generated locally with no network call to a provider.
• No data is logged by a third party or used to train future models.
• You can work fully offline once the model is downloaded.
• Sensitive documents, code, credentials, and personal data stay in your control.

Privacy guarantees hold only as far as your own setup: if you connect a local model to an app that also logs to an external service, that external logging may still capture data. Audit the full data path, not just the model.

Cost and offline use

Running a local model has no per-token API cost. After the one-time hardware investment, inference is effectively free. This matters for high-volume use cases: batch summarisation, automated code review pipelines, document processing, or any workflow that sends thousands of prompts.

Local models also work offline. Once the weights are downloaded, you can run the model on a plane, in a restricted network environment, or in a data centre with no outbound internet access.

Tradeoffs to understand before you start

• Quality ceiling: Most local models on consumer hardware are weaker than frontier hosted models on complex reasoning, very long context, and nuanced instruction following. Test your actual tasks.
• Setup overhead: Installing a runner, downloading model weights, configuring context and quantization, and debugging memory errors takes more effort than signing up for an API.
• Hardware dependency: Performance is tied to your hardware. Upgrading capability means upgrading hardware, not just switching a model ID.
• Model updates: You manage updates manually. Cloud providers update their models silently; with local models, pulling a new version is your responsibility.
• Memory pressure: Running a large local model alongside a browser, IDE, and other apps can cause memory contention. A dedicated inference machine removes this friction.

Getting started: the right order

1. Check your VRAM (GPU memory) and pick a model size that fits comfortably — leave 2 GB headroom.
2. Install Ollama from the official site and verify it runs with ollama --version.
3. Pull one small model: ollama pull qwen3:8b or ollama pull llama3:8b.
4. Run a quick test prompt: ollama run qwen3:8b "Explain local LLMs in three sentences."
5. Log the response speed, quality, and memory use before adding any other tools.
6. Only after the base model works reliably: add Open WebUI, a coding assistant, or a RAG layer.

Sources

FAQ