Dual RTX 3090 48GB for Local LLMs: The NVLink Path to 70B

What this hardware can run

With a working tensor-parallel runtime, the 48GB combined pool opens 70B models at Q4 and full-FP16 30B-class models — both out of reach for a single 24GB card. Without that runtime, you effectively have two separate 24GB cards, not a 48GB system.

A note on the 1872 GB/s figure: this is the sum of both cards' individual memory bandwidth (~936 GB/s each), not a single number that any one model transaction sees. Real throughput on a tensor-parallel 70B run depends on how efficiently the runtime splits the model and how much overhead the NVLink bridge adds for cross-GPU synchronization — expect meaningful but not literal 2x scaling versus a single card at the layers that fit.

Best model sizes for this hardware

• 70B at Q4_K_M: The reason to build this rig. Llama 3.1 70B or Qwen 2.5 72B at Q4_K_M (~40-42GB) fits inside the 48GB pool with a tensor-parallel runtime. Expect meaningfully faster generation than Apple Silicon 64GB at the same model, since GPU memory bandwidth per card is much higher than unified memory bandwidth.
• 30B-32B at Q8: Near-FP16 quality at 32B (~32-34GB) — a tier that does not fit on a single 24GB card. Good middle ground if 70B feels too slow for interactive use but you still want maximum quality at 32B.
• 13B-14B at FP16 on a single card: If you do not need the full 48GB pool for one model, you can run a 13B-14B model at FP16 on one card while leaving the second card free for a second model, embeddings, or a vision model — a flexible dual-GPU serving pattern that does not require NVLink at all.

Recommended models

• Llama 3.1 70B Q4_K_M — The flagship use case for this hardware. Requires a tensor-parallel runtime to split across both cards.
• Qwen 2.5 72B Q4_K_M: Comparable VRAM footprint to Llama 3.1 70B. Strong reasoning and long-context performance for a 48GB tensor-parallel setup.
• Qwen 2.5 32B Q8: A faster, still high-quality alternative to 70B when generation speed matters more than absolute model size.
• Two independent 13B models: If tensor parallelism is more complexity than you want yet, run one 13B-14B model per card independently — no NVLink coordination required, simple multi-model serving.

Getting started: first setup on dual RTX 3090s

Plain ollama pull and ollama run will not give you a working 70B model split cleanly across both cards with NVLink. The realistic path is vLLM with tensor parallelism, or a tensor-parallel build of llama.cpp. Below is a representative vLLM launch — adjust model name, tensor-parallel size, and quantization to your setup.

# Confirm both GPUs and NVLink are visible
nvidia-smi
nvidia-smi topo -m   # look for "NV" links between GPU0 and GPU1

# Install vLLM (Linux, CUDA required)
pip install vllm

# Launch a 70B model with tensor parallelism across both cards
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3.1-70B-Instruct \
  --quantization awq \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

# Alternative: llama.cpp built with tensor-parallel / multi-GPU support
# (build with CUDA + split-mode row for tensor-parallel-style splitting)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
./build/bin/llama-server \
  -m llama-3.1-70b-q4_k_m.gguf \
  --split-mode row \
  --tensor-split 1,1 \
  --n-gpu-layers 999

Note: --tensor-split in llama.cpp balances layers across GPUs but does not use NVLink for tensor-parallel compute the way vLLM does — for genuine NVLink-accelerated tensor parallelism, vLLM or GPUStack are the more mature options as of mid-2026.

Recommended runtimes

• vLLM: The most mature open-source option for NVLink-aware tensor-parallel serving. Set --tensor-parallel-size 2 to split a model across both cards. Best choice if you want an OpenAI-compatible API server for 70B-class models.
• GPUStack: A newer multi-GPU orchestration layer built for exactly this use case — distributing large models across heterogeneous or matched GPU clusters, including NVLink pairs. Worth evaluating alongside vLLM.
• llama.cpp (tensor parallel / row-split build): More manual to configure than vLLM, but works well for GGUF-quantized models if you are already invested in the llama.cpp ecosystem. Requires building with CUDA support and understanding split-mode flags.
• Ollama — Not recommended as the primary runtime for this hardware. It can offload layers across both GPUs, but it does not do NVLink tensor-parallel compute, so you lose most of the benefit of the NVLink bridge you paid for.

Best local AI workflows for this hardware

• Local 70B inference: The primary reason to build this rig. Use vLLM to stand up an OpenAI-compatible endpoint serving Llama 3.1 70B or Qwen 2.5 72B for personal or small-team use.
• High-throughput batch inference pipelines: vLLM's continuous batching combined with two GPUs makes this hardware well suited to processing large volumes of prompts — document summarization, dataset labeling, or bulk generation jobs — faster than a single card.
• Serving two models simultaneously: If you do not need tensor parallelism for a single giant model, run a coding model on one card and a chat or reasoning model on the other — a straightforward way to use 48GB without NVLink complexity.
• Fine-tuning smaller models: LoRA fine-tuning of 7B-13B models benefits from the extra VRAM headroom across two cards, even without full tensor-parallel training.

What this hardware cannot do well

• Beginner-friendly setup: This is explicitly an advanced path. If you have never configured CUDA, built software from source, or debugged GPU driver issues, start with a single card and a beginner-friendly runtime like Ollama first.
• Automatic VRAM pooling: NVLink does not magically give every application 48GB of usable memory. Software that is not NVLink/tensor-parallel-aware sees two separate 24GB devices, not one 48GB device.
• Low power draw: Two RTX 3090s under load draw roughly 700W combined, on top of the rest of the system. This requires a robust PSU (typically 1000W+), good case airflow, and a real electricity cost to factor in for sustained use.
• 70B at Q8 or FP16: Q8 on a 70B model needs roughly 74GB, and FP16 needs roughly 140GB — both exceed the 48GB pool. Q4_K_M is the practical ceiling for 70B on this hardware.

Upgrade path

• Dual RTX 4090 48GB: Higher combined bandwidth (~2016 GB/s) and no NVLink support on the RTX 4090 — meaning you are committed to PCIe-only tensor-parallel runtimes (vLLM, llama.cpp) rather than an NVLink bridge. Faster per-card, but the interconnect story is different since NVLink was dropped from the 40-series consumer lineup.
• 4x GPU workstation (96GB+): The natural next step for teams that need to serve multiple 70B-class models simultaneously or want headroom for long-context, high-batch serving. Significantly higher cost and power/cooling requirements.
• Data center GPUs (A100 80GB, H100): A single A100 80GB card holds a 70B model at Q8 or even FP16 without any multi-GPU coordination at all — simpler software story, but far more expensive per card than a used RTX 3090.

Cloud GPU fallback

If the setup friction of NVLink and tensor-parallel runtimes is more than you want to take on right now — or you need higher-precision 70B inference (Q8/FP16) than 48GB can hold — cloud GPU rentals are a practical way to test the same model class before committing to hardware.

Related hardware

FAQ

Disclosure

OpenSourcesAI may earn a commission or referral fee from links to hardware retailers, cloud GPU providers, or partner tools on this page. Editorial assessments are produced independently and are not influenced by commercial relationships. Hardware specs are sourced from manufacturer documentation. Model VRAM estimates are derived from GGUF quantization formulas and may vary across runtime versions and model architectures. NVLink tensor-parallel performance varies significantly by runtime, model architecture, and configuration. Verify before making purchasing decisions.

Last verified: July 2026