Hardware · NVIDIA Blackwell
RTX 5090 32GB for Local LLMs: The Flagship Consumer AI Card
The RTX 5090 32GB is the current flagship consumer GPU for local AI — the top of the single-GPU tier for anyone not moving to a Mac Studio or a multi-GPU workstation. 32GB of GDDR7 running at 1792 GB/s gives it comfortable headroom for 32B models at Q8, and it is the only consumer single card that can attempt a 70B model locally without Apple Silicon's unified memory approach. That 70B capability comes with a real caveat covered below — this is a fast, honest guide to what the card can and cannot do.
Editorial review
Bandwidth and pricing for newly released Blackwell-generation cards can shift as retail availability stabilizes. 70B offload performance varies by runtime version — verify current benchmarks before purchasing for a 70B-first workload.
Quick verdict
The fastest, highest-capacity single consumer GPU for local AI — near-perfect up to 32B at Q8, with 70B reachable only via offload assist.
- 7B–14BFull FP16 with large headroom
- 32BQ8 at ~30 GB — the recommended daily driver
- 70B Q4_K_M~41 GB — needs partial CPU/RAM offload
- 70B Q2/IQ2Fits fully but quality degrades noticeably
Good fit for
- Workloads at 32B and below that want the best single-GPU experience available
- Fast production-quality inference at 27B–32B Q8
Wrong fit for
- 70B as the primary goal — weigh Apple Silicon 64GB or a multi-GPU workstation first
The RTX 5090 32GB is the fastest and highest-capacity single consumer GPU currently available for local AI. Its real strength is 32B-class models at Q8 — comfortable, fast, production-quality inference at the highest single-GPU consumer tier. Its headline feature, 70B support, is real but constrained: 32GB alone is tight for Llama 3.3 70B at Q4_K_M (~41 GB), so a full 70B run needs partial CPU/RAM offload assist, which trades away some of the card's speed advantage.
If your workloads live at 32B and below, this card is close to as good as single-GPU local AI gets. If 70B is the primary goal rather than an occasional stretch, weigh it against Apple Silicon 64GB unified memory or a multi-GPU workstation before committing.
What this hardware can run
32GB opens up the entire sub-32B tier at high quantization with real headroom, and makes 70B reachable — with a caveat on how it gets there.
| Model size | Best quantization | VRAM used | Verdict | Notes |
|---|---|---|---|---|
| 7B–14B | FP16 | 14–28 GB | Comfortable | Full precision with large headroom, very fast generation. |
| 24B | Q8 | ~25.5 GB | Comfortable | Mistral Small 3.1 at Q8 — near-lossless quality with margin to spare. |
| 27B | Q8 | ~29 GB | Tight | Gemma 3 27B at Q8 fits, but with ~1.5 GB to spare after overhead — keep context moderate. |
| 32B | Q4_K_M | ~20 GB | Comfortable | Qwen3 32B / Qwen2.5 32B at Q4_K_M — the recommended daily driver. Their Q8 (~34 GB) exceeds 32GB and needs RAM offload. |
| 32B | FP16 | ~64 GB | Does not fit | Exceeds 32GB — Q8 is the practical ceiling for full-precision-adjacent quality. |
| 70B | Q4_K_M | ~41 GB | Tight — needs offload | Llama 3.3 70B exceeds 32GB alone. Runs with partial CPU/RAM offload at reduced speed. |
| 70B | Q2/IQ2 | ~26–29 GB | Fits, quality tradeoff | Very aggressive quantization can fit fully in VRAM, but output quality degrades noticeably. |
Generation speed reference: a 32B Q8 model runs at roughly 25–35 tokens per second on the RTX 5090 thanks to its 1792 GB/s bandwidth — noticeably faster than the same model on a 24GB Ada Lovelace card at a lower quant. A 70B model with partial offload drops well below this, since every layer that spills to system RAM adds latency the GPU alone would not have.
Best model sizes for this card
- 32B at Q8: The recommended daily driver. Near-lossless quality with real headroom for long context — this is where the RTX 5090 is unambiguously the best single-GPU consumer option available.
- 27B at Q8: Gemma 3 27B fits comfortably with margin to spare, useful when you want faster generation than the 32B tier without stepping down to a lower-quality quant.
- 70B at Q4_K_M (offload-assisted): Usable for occasional heavier reasoning tasks. Expect meaningfully slower generation than any model that fits fully in the 32GB VRAM budget — this is a capability, not a comfortable daily-driver setup.
Recommended models
- Qwen3 32B Q8_0: Pull with
ollama pull qwen3:32b-q8_0. The headline daily-driver model for this card. - Qwen2.5 32B Q8_0: Pull with
ollama pull qwen2.5:32b-q8_0. A strong alternative with a mature ecosystem of fine-tunes. - Gemma 3 27B Q8_0: Pull with
ollama pull gemma3:27b-q8_0. Fast, comfortable, and well-suited to general chat and instruction following. - Llama 3.3 70B Q4_K_M: Pull with
ollama pull llama3.3:70b-q4_K_M. Requires partial CPU/RAM offload on 32GB alone — set expectations for reduced speed before relying on it.
Getting started: first setup on the RTX 5090
# Install Ollama (Linux/macOS)
curl -fsSL https://ollama.com/install.sh | sh
# Confirm CUDA is detected
ollama serve &
ollama list
# Start with 27B Q8 — confirms CUDA is working with headroom to spare
ollama pull gemma3:27b-q8_0
ollama run gemma3:27b-q8_0
# Move to the recommended daily driver: 32B at Q8
ollama pull qwen3:32b-q8_0
ollama run qwen3:32b-q8_0
# Attempt 70B — expect partial offload and reduced speed on 32GB alone
ollama pull llama3.3:70b-q4_K_M
ollama run llama3.3:70b-q4_K_MRecommended runtimes
- Ollama — Detects CUDA automatically. Run
ollama pull qwen3:32b-q8_0to begin. - LM Studio — Desktop GUI for downloading, comparing, and multi-model serving without a terminal.
- vLLM: The RTX 5090's Blackwell architecture and bandwidth make it a strong single-GPU serving option for a small team via vLLM.
- GPUStack: Useful for multi-model serving setups where you want to run several smaller models alongside a primary 32B model within the 32GB budget.
Best local AI workflows for this card
- Production-quality daily chat driver: 32B at Q8 delivers quality and speed that competes with hosted APIs for most conversational and reasoning tasks.
- Multi-model serving: 32GB of headroom lets you keep a primary 32B model loaded alongside a smaller model for fast, simple requests, using LM Studio or GPUStack.
- High-speed local coding agents: A 32B or 27B model at Q8 gives strong code quality with generation speed fast enough for interactive agentic workflows.
- Occasional 70B reasoning tasks: When a task genuinely needs 70B-level depth, the card can deliver it with offload — just not as a default, always-on workflow.
What this hardware cannot do well
- 70B as a comfortable daily driver: 32GB alone is tight for Llama 3.3 70B at Q4_K_M (~41 GB). Full 70B use requires CPU/RAM offload assist, which meaningfully slows generation compared to a model that fits entirely in VRAM.
- Multi-GPU VRAM pooling: No NVLink support — running two RTX 5090s does not create a single larger address space. Multi-GPU setups need a tensor-parallel runtime instead.
- 70B at FP16 or near-lossless quants: Full-precision 70B requires well over 100GB of memory — entirely out of reach for a single 32GB card regardless of offload.
Upgrade path
- Apple Silicon 64GB+ unified memory: If 70B is a routine rather than occasional workload, a Mac with 64GB or more of unified memory fits Llama 3.3 70B at Q4 without offload — slower token generation than the 5090, but no VRAM ceiling to work around.
- Dual RTX 3090 / RTX 4090 workstation: Pairing two 24GB cards with a tensor-parallel runtime (vLLM, GPUStack) unlocks comfortable 70B inference with more headroom than a single 32GB card, at the cost of a much more complex setup.
Cloud GPU fallback
When a workload exceeds the 32GB ceiling of the RTX 5090 — full-precision 70B, fine-tuning runs, or batch inference at scale — these cloud GPU services provide on-demand access to larger VRAM pools.
Best cloud GPU for on-demand inference and spot rentals
RunPod
RunPod offers on-demand and spot GPU rentals with RTX 4090 (24 GB), A100 (80 GB), and H100 instances. A good fit for full-precision 70B runs or fine-tuning beyond this card's 32GB ceiling.
Pros
- Spot pricing makes large-model experiments affordable
- RTX 4090 and A100 instances available on demand
- Full-stack GPU pods with persistent storage and networking
Cons
- Spot instances can be interrupted mid-run
- Requires Docker familiarity for custom environments
Partner link: OpenSourcesAI may earn a commission if you sign up.
Visit RunPodBest cloud GPU for ML training and large-model fine-tuning
Lambda
Lambda Labs provides A100 and H100 GPU cloud instances purpose-built for ML workloads — a strong choice for fine-tuning runs beyond the RTX 5090's local 32GB capacity.
Pros
- A100 and H100 instances for serious ML workloads
- Clean pricing, no hidden fees, ML-focused infrastructure
- Strong fit for fine-tuning and batch inference jobs
Cons
- Less flexible than RunPod for short spot experiments
- GPU availability can vary by region and instance type
Related hardware
FAQ
Is the RTX 5090 32GB good for local AI?
Yes — it is the current flagship consumer GPU for local AI, and the only single consumer card that can attempt a 70B model locally without relying on Apple Silicon's unified memory trick. Its GDDR7 memory delivers 1792 GB/s of bandwidth, roughly 78% higher than the RTX 4090's 1008 GB/s, and 32GB of VRAM lets you run 32B models at Q8 with real headroom.
Can the RTX 5090 actually run a 70B model?
It can, but be realistic about the margin. Llama 3.3 70B at Q4_K_M weighs in around 41 GB, which does not fit in 32GB of VRAM alone. In practice this means partial CPU/RAM offload — a runtime like llama.cpp or Ollama keeps most layers on the GPU and spills the rest to system RAM. It works and is usable, but generation speed drops noticeably compared to a model that fits entirely in VRAM. If 70B is a core requirement rather than an occasional stretch, a 48GB+ card or Apple Silicon with 64GB+ unified memory is a more comfortable fit.
RTX 5090 vs Apple Silicon 64GB for 70B models — which is better?
Apple Silicon with 64GB of unified memory fits Llama 3.3 70B at Q4 entirely in memory, without offload — so it is more comfortable on paper. But the RTX 5090's raw compute and 1792 GB/s bandwidth make it dramatically faster at token generation on anything that fits its VRAM, and even with partial offload it often still generates faster than the Mac. Choose the 5090 if speed on 32B-and-under models matters most and 70B is occasional; choose Apple Silicon if 70B is a routine workload and you value quiet, low-power operation over raw throughput.
RTX 5090 vs RTX 4090 — is the upgrade worth it?
If you regularly work with 32B-class models or want to attempt 70B locally, yes — the extra 8GB of VRAM (32GB vs 24GB) changes what fits, not just how fast it runs. If your workloads stay comfortably under 24GB, the RTX 4090 remains an excellent card and the 5090's bandwidth advantage matters less once a model already fits with headroom on the 4090.
Does the RTX 5090 support NVLink for multi-GPU setups?
No. Like the RTX 4090 before it, the RTX 5090 has no NVLink support — it is PCIe only. Running two 5090s does not pool their VRAM into a single 64GB address space; multi-GPU inference requires a tensor-parallel runtime such as vLLM or GPUStack that splits the model across cards over PCIe, which is slower than true VRAM pooling.
What is the best first model to try on the RTX 5090?
Start with a 32B model at Q4_K_M to confirm the setup and see what the card is really capable of — this is its comfort zone, with ~8 GB of headroom to spare. Run `ollama pull qwen3:32b` first. For near-lossless quality, `mistral-small3.1:24b-instruct-2503-q8_0` runs a 24B at Q8 comfortably; note that 32B at Q8 (~34 GB of weights) exceeds 32GB and falls back to slow RAM offload.
Compare availability
Shopping links are optional and may be paid affiliate links. They never affect which hardware we recommend.
As an Amazon Associate I earn from qualifying purchases.
Disclosure
OpenSourcesAI may earn a commission or referral fee from links to hardware retailers, cloud GPU providers, or partner tools on this page. Editorial assessments are produced independently and are not influenced by commercial relationships. Hardware specs are sourced from manufacturer documentation. Model VRAM estimates are derived from GGUF quantization formulas and may vary across runtime versions and model architectures. Verify before making purchasing decisions.
Check model fit for your exact setup
Enter your VRAM, RAM, and workflow into the compatibility checker to get model recommendations matched to your specific hardware.
For builders
Selling flagship or enthusiast-tier AI hardware?
Sponsor a contextual placement on this page or submit your product for editorial review on OpenSourcesAI. For sponsorship options, advertise with us. For submissions or corrections, use the submit page.