Hardware tier · 12GB VRAM
What Can 12GB VRAM Run? Local AI at the 12GB Tier
12GB of GPU VRAM is the most common capable local AI tier. It is enough to run 7B and 8B models at Q8 quality and 13B–14B models at Q4 — covering the majority of practical open-weight model use cases. It cannot run 30B+ models at interactive speed without CPU offload, and it cannot run any model at FP16 above the 7B size class. This guide covers exactly what fits, which GPUs offer this tier, and when 24GB makes sense.
Editorial review
GPU pricing changes frequently. This page covers the 12GB VRAM tier as it applies across multiple GPU models. All 12GB GPUs share the same model ceiling — bandwidth differences affect tokens-per-second, not which models fit. Verify current pricing before purchasing.
Verdict: 12GB VRAM Is the Practical Entry Point for Local AI
The default starting point for consumer local AI — strong 7B and 8B models with no cloud dependency, with 14B at Q4 as the hard ceiling.
- 7B–8BQ8 with 3–4 GB headroom — the daily tier
- 13B–14BQ4_K_M; Q8 (~14 GB) does not fit
- 30B+CPU offload only at 1–5 t/s — needs 24GB+
- Speed spreadRTX 3060 at 360 GB/s vs RTX 4070 Ti at 504 GB/s — same ceiling, different t/s
Good fit for
- First local AI builds — the sensible budget floor
- Private chat, coding assistance, and light RAG at 7B–14B
Wrong fit for
- 30B-class models at interactive speed
- Q8 quality on 13B+ — that starts at the 24GB tier
The 12GB VRAM tier is the default starting point for consumer local AI because it can run strong 7B and 8B models without relying on cloud infrastructure. Performance depends heavily on bandwidth: an RTX 3060 at 360 GB/s generates noticeably fewer tokens per second than an RTX 4070 Ti at 504 GB/s on the same model. The hard ceiling is 14B at Q4 quantization; anything larger requires CPU offload, a 24GB GPU, or cloud fallback.
Model fit at 12GB VRAM
The table below applies to any GPU with 12GB of VRAM. Individual cards differ in token generation speed (memory bandwidth), but the model ceiling is the same across all 12GB GPUs.
| Model size | Best quantization | VRAM used | Fits in 12GB? | Notes |
|---|---|---|---|---|
| 1B–4B | FP16, Q8, or Q4 | 0.7–8 GB | Yes | Any quantization. Good for fast tools and experimentation. |
| 7B | Q8 (best on 12GB) | ~7.7 GB | Yes | Q8 fits with ~4GB headroom. FP16 requires ~14GB — does not fit. |
| 8B | Q8 | ~8.7 GB | Yes (short context) | Llama 3 8B, Gemma 3 8B. Measured (Qwen3 8B, RTX 3080 10GB): fits fully in VRAM at 4K context; needs 16K before Ollama splits to system RAM. |
| 13B | Q4_K_M | ~7.9 GB | Yes | Q4 fits. Q8 (~14 GB) does not fit at 12GB. |
| 14B | Q4_K_M | ~9 GB | Yes | Qwen 2.5 14B, Phi-4. Q4 fits with about 3 GB headroom — enough to stay on the card, unlike a 10GB GPU. |
| 30B–32B | Q4 | ~18–20 GB | CPU offload only | Exceeds 12 GB. RAM offload runs at 1–5 t/s. Needs 24GB+ for interactive use. |
| 70B | Q4 | ~42–47 GB | No | Far exceeds 12 GB. Needs cloud or workstation hardware. |
Use the Local LLM Compatibility Checker to match specific models against your exact hardware configuration and workflow.
What 12GB VRAM enables
- 7B and 8B at Q8: The best quality level available at 12GB for these model sizes. Q8 is very close to FP16 quality — the difference is minimal for most tasks. Runs at 15–40 tokens per second depending on the GPU bandwidth. Measured on an RTX 3080 10GB: fits fully in VRAM through a 4K context; long coding or RAG sessions pushing toward 16K start splitting the model onto system RAM, so watch context length rather than treating the fit as unconditional.
- 14B at Q4_K_M: Larger model depth at reduced precision. Good for tasks where model knowledge matters more than maximum quantization quality. Fits with about 3 GB headroom.
- Fast 4B models at any quantization: FP16 on 4B models fits well under 12GB. Very fast inference — ideal for tools, autocomplete agents, and lightweight coding assistants.
What 12GB VRAM cannot do
- 7B or 8B at FP16: FP16 on 7B needs ~14GB. FP16 on 8B needs ~16GB. Both exceed 12GB. Q8 is the best available quality at this VRAM level for these model sizes.
- 13B or 14B at Q8: Q8 on 13B requires ~14GB. Q8 on 14B requires ~15GB. Both exceed 12GB. Q4 is the best available quality for 13B+ on this tier.
- 30B+ at interactive speed: Q4 on 30B models needs 18–20GB. CPU offload is possible but generates at 1–5 tokens per second — not suitable for interactive chat.
GPUs at the 12GB tier: architecture constraints
All 12GB GPUs have the same model ceiling. The meaningful difference is memory bandwidth, which determines how many tokens per second you get at each model size.
| GPU | Bandwidth | Architecture | Notes |
|---|---|---|---|
| RTX 3060 12GB | 360 GB/s | Ampere | Budget entry. Same model ceiling, lower bandwidth — ~30% fewer t/s than the 4070 Ti at the same model. |
| RTX 4070 12GB | 504 GB/s | Ada Lovelace | Mid-range Ada Lovelace at 12GB. Good all-around local AI card. Similar bandwidth to 4070 Ti. |
| RTX 4070 Ti 12GB | 504 GB/s | Ada Lovelace | Prosumer. Highest bandwidth at the 12GB tier — fastest tokens-per-second for 12GB cards. |
| RTX 3080 12GB | 912 GB/s | Ampere | Exceptionally high bandwidth at 12GB. Less common; strong used-market value if available. |
- Ada Lovelace (RTX 4070, RTX 4070 Ti): NVIDIA's current generation. Better power efficiency, higher tensor core performance, and more bandwidth-per-watt than Ampere. The RTX 4070 Ti at 504 GB/s is the fastest consumer 12GB option. Both run Ollama, LM Studio, and vLLM without modification on CUDA 12.x.
- Ampere (RTX 3060, RTX 3080 12GB): Previous-generation consumer architecture. The RTX 3060 at 360 GB/s is the budget entry — same model ceiling as the 4070 Ti but ~30% slower at the same model size. The RTX 3080 12GB is unusual: it has very high bandwidth (912 GB/s) at 12GB, making it a strong value if found at the right used-market price.
Rule of thumb: more bandwidth = more tokens per second for the same model. The RTX 4070 Ti at 504 GB/s generates roughly 30–40% more tokens per second than the RTX 3060 at 360 GB/s on the same model and quantization level.
Recommended models for 12GB VRAM
- Qwen 3 8B Q8 — daily driver:
ollama pull qwen3:8b-q8_0. Near-FP16 quality on a strong 8B model. Measured at ~8.7 GB fully in VRAM at a 4K context; give it room for longer coding or RAG sessions rather than assuming unlimited headroom. - Qwen 2.5 14B Q4_K_M — depth over quality:
ollama pull qwen2.5:14b. A 14B model at Q4 for more model knowledge depth. Useful when model size matters more than quantization precision. - Gemma 3 4B FP16 — fast and precise:
ollama pull gemma3:4b-fp16. Full precision on a small model. Very fast inference for autocomplete, tools, and lightweight agents. - Phi-4 Q4_K_M — efficient 14B reasoning:
ollama pull phi4. Strong reasoning per parameter. Fits within 12GB at Q4 with headroom for moderate context windows.
Getting started: first setup on a 12GB GPU
Start with Ollama and a 7B or 8B model to verify the CUDA stack works, then upgrade to Q8:
# Install Ollama (Linux/macOS)
curl -fsSL https://ollama.com/install.sh | sh
# Pull a 7B model at default Q4 first (fast download, confirms CUDA is working)
ollama pull qwen3:8b
# Once working, upgrade to Q8 for better quality (~8.7 GB VRAM)
ollama pull qwen3:8b-q8_0
ollama run qwen3:8b-q8_0
# For more model depth: 14B at Q4
ollama pull qwen2.5:14b
ollama run qwen2.5:14bBest local AI workflows for this tier
- Private local chat assistant: A 7B or 8B model at Q8 runs fast enough for conversational use. No cloud dependency, no data leaving your machine.
- Code completion and review: 7B and 8B models tuned for code (Qwen 2.5 Coder, DeepSeek-Coder) run well at Q8 on 12GB. Fast enough for inline completions with short latency.
- Document chat (RAG): 7B or 8B models work well for retrieval-augmented generation over small to mid-size document sets. Pair with a lightweight embedding model and a local vector store like Qdrant or Chroma.
- Lightweight agent workflows: 4B models at FP16 or Q8 are fast enough for tool-call loops in agent frameworks. 14B at Q4 adds more reasoning depth for complex chains.
When to upgrade to 24GB
The 24GB tier unlocks three things the 12GB tier cannot do:
- 7B and 8B at FP16: Full precision inference with no quality loss from quantization.
- 13B and 14B at Q8: Near-lossless quality on larger models.
- 30B and 32B at Q4: Model depth that requires 18–20GB — genuinely impossible at 12GB at interactive speed.
If your daily workflows fit within 7B Q8 or 14B Q4, the 12GB tier is sufficient. The upgrade to 24GB is most justified when you regularly hit the 13B Q4 quality ceiling and want Q8, or when 30B model depth is a hard requirement.
Cloud GPU fallback
A 12GB GPU is excellent for private assistants, lightweight coding helpers, document chat, and local experimentation. Once a workflow requires 30B or 70B models, higher concurrency, or long context windows that exhaust the KV cache headroom, cloud GPU scaling is the practical fallback.
Best cloud GPU for on-demand inference and spot rentals
RunPod
RunPod offers A100 80GB and RTX 4090 cloud instances on demand and at spot pricing. The right choice when 30B or 70B workloads exceed the local 12GB ceiling.
Pros
- RTX 4090 24GB for immediate 30B model access
- A100 80GB for 70B at Q8 — no quality compromise
- Spot pricing for budget-sensitive experiments
Cons
- Spot instances can be interrupted mid-run
- Requires Docker familiarity for custom environments
Partner link: OpenSourcesAI may earn a commission if you sign up.
Visit RunPodRelated hardware
FAQ
What is the best model size for a 12GB VRAM GPU?
For most workflows, 7B and 8B models at Q8 quantization are the sweet spot at 12GB. They use roughly 7.7–8.7 GB of VRAM at a short context and run at 15–40 tokens per second depending on the GPU — measured on an RTX 3080 10GB, Qwen3 8B Q8_0 loaded fully into VRAM at a 4K context, but needed 16K of context before Ollama started splitting the model between VRAM and system RAM. So Q8 at this size is genuinely comfortable for chat-length conversations, and worth watching if you push long coding or RAG context. For tasks where model depth matters more than speed, 13B models at Q4_K_M fit comfortably at about 7.9 GB. A 14B is tighter than it looks: measured weights run about 9 GB, which still fits a 12GB card but leaves roughly 3 GB for context rather than the 3.5 GB an 8.4 GB estimate implies. FP16 on 7B or 8B models does not fit because it requires 14–16 GB.
Can 12GB VRAM run 30B models?
Only with CPU offload, and at very slow speeds. A 30B model at Q4 needs approximately 18–20 GB of VRAM. On a 12GB GPU, the model layers that do not fit in VRAM are offloaded to system RAM and processed by the CPU — this produces roughly 1–5 tokens per second, which is too slow for interactive chat. For 30B models at conversational speed, you need a 24GB GPU or cloud inference.
What is the difference between the RTX 3060 12GB and RTX 4070 Ti 12GB for local AI?
Both have 12GB of VRAM, so the model ceiling is identical — the same models fit on both cards. The key difference is memory bandwidth: the RTX 4070 Ti has 504 GB/s versus the RTX 3060's 360 GB/s, which produces roughly 30–40% more tokens per second at the same model and quantization level. The RTX 3060 is the budget entry point; the RTX 4070 Ti is the fastest 12GB option but costs significantly more.
Why does quantization matter so much at 12GB VRAM?
At 12GB, every gigabyte counts. A 7B model at FP16 (no quantization) requires about 14 GB or more — it simply does not fit. The same model at Q8 measures about 9 GB of weights and fits, though tightly once runtime overhead is counted. Q4 on 7B uses about 5–6 GB and fits with room to spare. Quantization is the tool that makes larger models fit inside a small VRAM budget, with a quality tradeoff: Q8 is very close to FP16, while Q4 has a small but perceptible quality reduction on nuanced tasks.
When does it make sense to upgrade from 12GB to 24GB VRAM?
Upgrade when any of these apply: you want 7B or 8B models at FP16 precision (not just Q8), you regularly use 13B+ models and the Q4 quality ceiling feels limiting, or you need 30B and 32B models at interactive speed. If your daily workflows fit within 7B Q8 or 14B Q4, the 12GB tier is sufficient and the cost of upgrading to 24GB may not be justified.
Compare availability
Shopping links are optional and may be paid affiliate links. They never affect which hardware we recommend.
As an Amazon Associate I earn from qualifying purchases.
Disclosure
OpenSourcesAI may earn a commission or referral fee from links to cloud GPU providers or partner tools on this page. Editorial assessments are produced independently and are not influenced by commercial relationships. Hardware specs are sourced from manufacturer documentation. Model VRAM estimates are derived from GGUF quantization formulas and may vary across runtime versions and model architectures. Verify before making purchasing decisions.
Check your specific GPU against models
Enter your VRAM, RAM, and workflow into the compatibility checker to see which models and quantization levels are recommended for your 12GB setup.
For builders
Selling AI hardware, cloud GPUs, or local AI tools?
Sponsor a contextual placement on this page or submit your product for editorial review on OpenSourcesAI. For sponsorship options, advertise with us. For submissions or corrections, use the submit page.