Stack recipe · Reviewed June 2026

Voice AI Assistant Stack

A voice-in, voice-out AI pipeline combining Whisper for transcription, a local LLM for reasoning, and ElevenLabs or Coqui TTS for speech synthesis — no cloud transcription required.

Best for

Builders prototyping voice-controlled AI assistants, accessibility tools, podcast transcription pipelines, or speech-driven workflows where audio never leaves the local environment.

Core tools

  • Whisper (OpenAI)
  • Ollama
  • ElevenLabs
  • Coqui TTS
  • Open WebUI

Recommended models

  • Qwen3 8B — fast reasoning, low latency
  • Phi-4 — compact with strong instruction following
  • Mistral Small 3.1 — reliable for structured voice response generation
  • Whisper large-v3 for transcription (run via Ollama or standalone)

Hardware notes

8–16 GB VRAM covers Whisper + a small LLM simultaneously. Whisper large-v3 alone needs about 3 GB VRAM; the chat model needs 5–16 GB depending on size. For CPU-only, Whisper still runs well; LLM inference will be slower.

Setup steps

  1. Install Ollama and pull a small fast chat model: ollama pull qwen3:8b
  2. Install Whisper for transcription: pip install openai-whisper (standalone) or use the whisper Ollama model.
  3. Record or stream audio input, pass it to Whisper to get a transcript.
  4. Send the transcript as a prompt to your local LLM via the Ollama API at localhost:11434.
  5. Pass the model response to ElevenLabs (cloud TTS with high quality) or Coqui TTS (local, open-source) for voice output.
  6. Wire the pipeline together with a short Python script or n8n workflow, then test end-to-end latency.
  7. Measure round-trip time: transcription + inference + synthesis should reach under 3 seconds on modern hardware.

Trade-offs

Local Whisper transcription is accurate and private but slower than cloud ASR. ElevenLabs produces higher-quality voices than Coqui but sends audio text to a cloud API. For fully private voice AI, use Coqui TTS and accept lower voice quality.

Alternatives

  • Use Deepgram or AssemblyAI for faster cloud transcription when privacy is not a constraint.
  • Use Piper TTS for a faster, lighter local TTS alternative to Coqui.
  • Use Open WebUI voice mode for a simpler no-code voice pipeline with Ollama.

Related resources

Ollama tool profileElevenLabs TTS tool profileOpen WebUI tool profilen8n tool profileAI Automation Workflow

Related stack recipes

Private Local Chatbot StackA practical stack for running private chat over local or self-hosted models with a browser chat UI and clear review habits.Local Coding Assistant StackA coding stack for testing open coding assistants with local or self-hosted models, repository-aware tools, and human code review.AI Automation Workflow StackA workflow automation stack for turning AI outputs into drafts, approvals, notifications, and controlled actions.

Not sure if your PC has enough VRAM for this workflow?

Run the Local LLM Hardware Checker →

FAQ

Can I run the entire voice pipeline locally without any cloud APIs?

Yes. Use Whisper standalone for transcription, Ollama for inference, and Coqui TTS or Piper for synthesis. All three run locally. Voice quality from Coqui is lower than ElevenLabs but is improving rapidly.

What is the realistic end-to-end latency for a voice assistant?

On a machine with a mid-range GPU (RTX 3090 or better), expect 1.5–3 seconds round trip: 0.3–0.8s transcription, 0.5–1.5s inference, 0.3–0.8s synthesis. CPU-only setups will be slower — 5–15 seconds is typical.

Get practical stack updates

Join the OpenSourcesAI update list for new stack recipes, tool notes, and developer-first comparisons.

Get updatesBrowse guides