Guide
What Is an AI Stack?
There is no single program called “local AI”. You assemble two to five separate pieces, each doing one job, and the assembled set is called a stack. This guide names the layers, shows which ones you can skip, and gives you the smallest stack that genuinely works.
Editorial review
AI tools, model releases, pricing, licenses, and platform terms can change quickly. Verify the official source before production or commercial use.
The short answer
A stack is the set of separate programs you run together to get a working AI setup.
This trips people up because most software arrives as one thing you install. Local AI does not. You install a program that runs models, then download a model for it to run, then usually add something to type into. Three downloads, three jobs, one working setup — and the word for that arrangement is a stack.
The reason it works this way is that each piece is genuinely interchangeable. The program that runs models does not care which model you give it. The chat window does not care which program is behind it. Keeping them separate means you can replace any one of them without touching the others, which is exactly what you want when a better model comes out next month.
The five layers
Almost every local AI setup you will ever read about is some subset of these five layers, in this order. Only the first two are required.
| Layer | Its one job | Common choices | Needed? |
|---|---|---|---|
| Model | Holds what the AI knows | Llama, Qwen, Mistral, Gemma, DeepSeek | Always |
| Runtime | Loads the model and generates text | Ollama, LM Studio, llama.cpp, vLLM | Always |
| Interface | Gives you somewhere to type | Open WebUI, a terminal, an IDE plugin | Usually |
| Retrieval | Lets the model read your documents | Qdrant, Chroma, pgvector + an embedding model | Optional |
| Tools | Lets the model do things, not just talk | MCP servers, agent frameworks | Optional |
Read that table top to bottom and you have the whole map. Most confusion about local AI is really confusion about which layer a particular name belongs to — and the tool directory is organized the same way for that reason.
The smallest stack that works
Two layers. Install Ollama, then run:
ollama run llama3.2:3bThat command downloads the model (layer one), starts the runtime (layer two), and drops you into a prompt. You are now running local AI. There is no interface layer because the terminal is doing that job, no retrieval because you have not asked it to read anything, and no tools because it is only being asked to talk.
This is worth doing before anything else, even if a terminal is not where you want to end up. It proves your hardware works, it takes about five minutes, and every more elaborate stack is this one with pieces added.
Three real stacks
Here is what the layers look like filled in, at three levels of ambition. Each of these has a full recipe in the stack directory.
- Private chat (3 layers). Ollama runs the model, a general-purpose model like Llama or Qwen provides the intelligence, and Open WebUI gives you a browser chat window that looks like the hosted tools you are used to. This is the stack most people actually want. See the recipe →
- Coding assistant (3 layers). Same runtime, but a code-tuned model, and the interface is a plugin inside your editor rather than a separate window. The stack barely changes; the interface layer moves into VS Code. See the recipe →
- Document search (5 layers). Everything above, plus an embedding model and a vector database so the AI can answer from your own files, plus the glue that fetches the right passages at question time. This is where the retrieval layer earns its place. See the recipe →
Notice that the jump from the first to the second stack is a different model, and nothing else. That is the layered structure paying off.
Four ways a first stack goes wrong
- Building all five layers at once. A vector database added before plain chat is reliable means that when something breaks, you have five suspects instead of one. Add one layer at a time and confirm each works.
- Choosing the model first. The model has to fit in your GPU memory, so your hardware decides the shortlist, not the other way around. The compatibility checker settles this in about a minute.
- Assuming a stack means containers. Docker is one convenient way to run several services. It is not part of the definition, and a two-layer stack needs none of it.
- Adding retrieval to fix a knowledge problem. Retrieval gives the model access to your documents. It does not make a small model reason better. If answers are weak on general questions, the model is the layer to change.
For builders
Built an AI tool or open-source project?
Submit it for review or sponsor a featured placement on OpenSourcesAI. For sponsorship options, advertise with us. For submissions or corrections, use the submit page.
Frequently asked questions
Is an AI stack the same as a tech stack in web development?
Same idea, different pieces. A web tech stack names the layers of a website — database, backend, frontend. An AI stack names the layers of a working AI setup — model, runtime, interface, and optionally retrieval and tools. In both cases the word just means "the set of components running together", and in both cases each layer is swappable without rebuilding the others.
What is the smallest stack that actually works?
Two layers: a runtime and a model. Install Ollama, run `ollama run llama3.2:3b`, and you have a complete working AI stack talking to you in a terminal. Everything else — a web interface, document search, tool access — is an addition you make once the first two layers are stable.
Do I need Docker to run an AI stack?
No. Ollama and LM Studio are ordinary desktop installers with no containers involved. Docker becomes useful once you add a vector database or a self-hosted web interface, because it saves you from installing several services by hand — but plenty of people run a two-layer stack for months without ever touching it.
Can I change one layer without rebuilding the rest?
Yes, and that is the main reason the layered structure is worth understanding. Swapping the model is a one-line change and your interface never notices. Swapping the interface leaves the model untouched. The layers talk to each other over a stable API — most local runtimes expose an OpenAI-compatible endpoint — so replacing a piece rarely disturbs its neighbours.
How is a stack different from a model?
A model is a file of weights. It cannot do anything by itself — something has to load it into memory and feed it text. That something is the runtime, and the model plus the runtime plus whatever you type into is the stack. People often say "I am running Llama" when they mean "I am running Llama through Ollama in Open WebUI" — three layers, one sentence.
Which layer should I choose first?
None of them — check your hardware first. Your available GPU memory decides which model sizes are realistic, and the model size then narrows your sensible runtime choices. Picking a model you like the sound of and discovering afterwards that it does not fit is the single most common way a first local AI setup stalls.
Next step: pick a stack and build it
The stack directory has step-by-step recipes with the exact commands for each layer. If you are not yet sure which model sizes your machine can hold, start with the compatibility checker — it decides the model layer for you, and the rest follows.