What Is VRAM and How Much Do You Need for Local AI?

Editorial review

AI tools, model releases, pricing, licenses, and platform terms can change quickly. Verify the official source before production or commercial use.

What VRAM is, in plain English

VRAM (Video RAM) is the dedicated memory built into your GPU. It is separate from your computer's system RAM. When a local LLM runs on GPU, the model weights — the billions of numerical parameters that define what the model knows — must be loaded into VRAM before any text can be generated.

If the model fits entirely in VRAM, generation is fast: typically 15–50 tokens per second on consumer GPUs. If the model is too large to fit, the runtime offloads layers to system RAM and runs them on the CPU instead. This is called CPU offload. It works, but it can be 5–20 times slower than full GPU inference.

This is why VRAM is the number you need to know before choosing a model. It is not about total system RAM, not about internet speed, and not about CPU speed for most inference workloads. It is about how many gigabytes your GPU has.

How to check your VRAM

Run one of these commands to see how much VRAM your GPU has:

# Windows — PowerShell
(Get-WmiObject Win32_VideoController).AdapterRAM / 1GB

# Windows — NVIDIA only
nvidia-smi --query-gpu=memory.total --format=csv,noheader

# Linux
nvidia-smi

# macOS (Apple Silicon — shows total unified memory, not separate VRAM)
system_profiler SPDisplaysDataType | grep "Total Number"

On Windows, you can also open Task Manager → Performance → GPU and read the Dedicated GPU Memory value. On NVIDIA systems, nvidia-smi is the authoritative source.

The VRAM formula for model weights

The minimum VRAM to load model weights follows a straightforward formula:

VRAM for weights (GB) ≈ (parameters_in_billions × bits_per_parameter) / 8

Examples:
  7B model at FP16 (16 bits):  (7 × 16) / 8  = 14 GB
  7B model at Q8  ( 8 bits):   (7 ×  8) / 8  =  7 GB
  7B model at Q4  ( 4 bits):   (7 ×  4) / 8  =  3.5 GB (+ ~0.5 GB overhead ≈ 4.1 GB)
 14B model at Q4  ( 4 bits):  (14 ×  4) / 8  =  7 GB   (+ overhead ≈ 8.4 GB)
 70B model at Q4  ( 4 bits):  (70 ×  4) / 8  = 35 GB   (+ overhead ≈ 38–42 GB)

Add 10–20% for runtime overhead, the KV cache, and activations. In practice, always leave at least 1–2 GB of VRAM headroom beyond the raw weight estimate. A model that exactly fills VRAM will likely crash or throttle under load.

KV cache: the hidden VRAM cost

Beyond the model weights, the KV cache consumes additional VRAM. The KV cache stores intermediate attention computation results so the model does not have to reprocess past tokens on every generation step.

KV cache size (bytes) =
  2 × n_layers × n_kv_heads × head_dim × context_length × bytes_per_element

Where:
  2           = one key tensor + one value tensor
  n_kv_heads  = number of KV heads (reduced in GQA models — often 8 for 7B models)
  head_dim    = per-head hidden dimension (typically 128)
  bytes       = 2 for FP16, 1 for INT8

Example — Llama 3 8B at 4K context (FP16 KV):
  2 × 32 layers × 8 KV heads × 128 × 4096 tokens × 2 bytes
  = ~0.54 GB of KV cache

Example — same model at 32K context:
  2 × 32 × 8 × 128 × 32768 × 2
  = ~4.3 GB of KV cache

At short context lengths (2K–4K), KV cache overhead is modest. At long context (16K–128K), it becomes a major budget item. If you plan to use large context windows, factor KV cache into your VRAM plan before choosing a model or quantization level.

Total VRAM budget estimate

Total VRAM ≈ model_weights + kv_cache + runtime_overhead

Conservative planning formula:
  Total ≈ (params_B × bits_per_param / 8) × 1.15  +  kv_cache_GB  +  1 GB

Example — 14B model, Q4_K_M, 8K context:
  Weights:  14 × 4 / 8 = 7 GB × 1.15 ≈ 8.1 GB
  KV cache: ~1 GB (at 8K context, GQA model)
  Overhead: 1 GB
  Total:    ~10.1 GB → needs 12 GB VRAM

Quantization formats and quality tradeoffs

Quantization reduces the numerical precision of model weights to lower memory use. GGUF format (used by Ollama, LM Studio, and llama.cpp) supports a range of quantization levels. For most beginners, Q4_K_M is the best starting point.

The _K_M and _K_S suffixes in GGUF names refer to the k-quant method variant. K-quants use a per-block mixed precision approach that improves quality at the same bit count compared to older uniform quants. Prefer Q4_K_M over Q4_0when both are available.

Model size vs VRAM reference table

Approximate VRAM needed per model size at common quantization levels, including a typical runtime overhead buffer. Actual values vary by model architecture, context window, and runtime. Always verify with the model card and your target runner.

Hardware tiers: what fits where

Apple Silicon: unified memory explained

Apple's M-series chips use a unified memory architecture: CPU and GPU share the same physical memory pool rather than having separate DRAM chips. For local AI, this means:

• A MacBook Pro M3 Max with 64 GB of unified memory can load models far larger than a 24 GB NVIDIA GPU allows.
• Memory bandwidth is the main constraint, not total capacity. Apple Silicon has very high bandwidth (~800 GB/s on M3 Max) but it is still lower than high-end NVIDIA GPUs at dedicated tasks.
• Throughput per GB of memory is generally lower than dedicated VRAM — Apple Silicon generates fewer tokens per second per GB than an equivalent NVIDIA setup.
• The larger addressable pool is still a real advantage for fitting large models that simply cannot run on consumer NVIDIA GPUs.
• Tools like Ollama and LM Studio support Apple Silicon via Metal acceleration on macOS.

What to do when you run out of VRAM

• Use a smaller quantization level: switch from FP16 to Q8, or from Q8 to Q4_K_M.
• Use a smaller model: a 7B Q4 running at 30 t/s beats a 14B model partially offloaded to CPU at 4 t/s for most interactive tasks.
• Reduce context window: lower --ctx-size in Ollama or the context slider in LM Studio. Halving the context halves the KV cache budget.
• Close VRAM-consuming apps: games, video software, and other ML processes compete for the same GPU memory.
• Enable CPU offload deliberately: some runtimes let you specify how many layers run on GPU vs CPU. Partial offload is better than no GPU at all.
• Consider a cloud GPU: for occasional large-model inference, renting a GPU via RunPod, Vast.ai, or similar services is often cheaper than upgrading hardware.

Quick hardware planning checklist

1. Check your GPU VRAM using nvidia-smi or Task Manager → GPU.
2. Find your target model's parameter count on its model card or Ollama library page.
3. Estimate weight VRAM: (params_B × bits / 8) × 1.15.
4. Add KV cache estimate for your planned context window.
5. Leave 1–2 GB headroom. If the total exceeds your VRAM, reduce quantization or choose a smaller model.
6. Test with Ollama or LM Studio. Check actual VRAM use with nvidia-smi dmon during inference.

Sources

FAQ