Stack recipe · Reviewed June 2026

Edge and Low-Power AI Stack

A minimal always-on inference setup for sub-4B models on Raspberry Pi 5, ARM SBCs, Intel NUCs, or embedded appliances with 8–16 GB RAM and no discrete GPU.

Best for

Embedded AI builders, homelab operators, and IoT integrators who need persistent local inference on low-power hardware — home automation triggers, offline Q&A endpoints, or monitoring alerts.

Core tools

  • Ollama
  • llama.cpp
  • Jan

Recommended models

  • Qwen3 1.7B Q4 — 1.4 GB RAM, usable for simple tasks on Pi 5
  • Phi-4 Mini Q4 — 2.5 GB RAM, strong reasoning in a compact package
  • Gemma 3 1B Q4 — sub-1 GB RAM, very fast on ARM
  • SmolLM2 1.7B — designed for edge inference, low memory footprint

Hardware notes

Raspberry Pi 5 (8 GB RAM) is the minimum recommended edge device for sub-2B models. Intel NUC or Beelink mini PC with 16 GB RAM handles up to 4B models comfortably. No discrete GPU required — all inference runs on CPU. Power draw: 5–25W depending on device.

Setup steps

  1. Install Ollama on ARM Linux: curl -fsSL https://ollama.ai/install.sh | sh (works on Pi 5 with 64-bit OS)
  2. Pull the smallest viable model: ollama pull qwen3:1.7b
  3. Test inference speed: ollama run qwen3:1.7b "Summarize: The quick brown fox..."
  4. Expose the Ollama API on your LAN: OLLAMA_HOST=0.0.0.0:11434 ollama serve
  5. Set up as a systemd service for always-on operation: create /etc/systemd/system/ollama.service
  6. Connect downstream apps (Home Assistant, n8n, or custom scripts) to http://<pi-ip>:11434
  7. Monitor RAM usage — keep model VRAM below 70% of total RAM to avoid swap thrashing.

Trade-offs

Edge inference is slow — well below interactive chat speed on ARM CPUs, though usable for short, non-interactive tasks like summaries or automation triggers. Only sub-4B models are practical. Context length should be kept short (1K–4K tokens) to control RAM usage. For faster edge inference, consider Jetson Orin (GPU on an embedded board) at higher cost and power.

Alternatives

  • Use NVIDIA Jetson Orin for edge inference with a real GPU at 10–20W power budget.
  • Use a cloud API with caching for latency-tolerant applications where inference quality matters more than local control.
  • Use llama.cpp directly (without Ollama) for maximum control over threading and memory mapping on constrained hardware.

Related resources

Ollama tool profileLocal LLM Compatibility CheckerLightweight Laptop AI stackWhat IS QuantizationWhat IS Vram Local AI

Related stack recipes

Private Local Chatbot StackA practical stack for running private chat over local or self-hosted models with a browser chat UI and clear review habits.Local Coding Assistant StackA coding stack for testing open coding assistants with local or self-hosted models, repository-aware tools, and human code review.Home Lab AI StackA home-lab stack for local inference, private chat, and optional smart-home integrations.

Not sure if your PC has enough VRAM for this workflow?

Run the Local LLM Hardware Checker →

FAQ

Can a Raspberry Pi 5 actually run a useful LLM?

Yes, for simple tasks. Small 1-2B models run slowly but usably on a Pi 5 with 8 GB RAM — good for home automation triggers, short summaries, and offline Q&A. It is not fast enough for interactive chat. For better edge performance, an Intel N100 mini PC with 16 GB RAM is a significant upgrade.

What is the minimum RAM for edge AI inference?

The absolute minimum is 8 GB RAM for a sub-2B model with a small context window. 16 GB RAM gives you access to 3B–4B models and more reliable operation under load.

Get practical stack updates

Join the OpenSourcesAI update list for new stack recipes, tool notes, and developer-first comparisons.

Get updatesBrowse guides