How to Choose a GPU for AI and Machine Learning (2026)

How to Choose a GPU for AI and Machine Learning (2026)

Photo by Alex Kotliarskyi on Unsplash

Quick Answer: The right GPU for AI in 2026 depends on your model size and workload type. For running 7B-13B LLMs, get an RTX 5090 (32GB) or used RTX 3090 (24GB). For 70B+ models, you need H100 cloud instances or multiple consumer GPUs. The most important spec is VRAM — it determines what models you can run. Tensor Cores are essential for any AI work. Don't buy a GPU without Tensor Cores for deep learning.

What Makes a Good AI GPU?

Five specs matter for AI workloads, in order of importance:

1. VRAM Capacity (Most Important)

VRAM determines what models you can load. No amount of speed helps if the model doesn't fit.

VRAM	What Fits
8GB	Small models (3B INT4)
12GB	7B INT4 with some context
16GB	7B INT4 with ample context, Mixtral INT4
24GB	13B INT4, 7B FP8, 34B INT4
32GB	70B INT4 (tight), 13B FP8, 34B INT4
48GB	70B INT4 comfortably, 34B FP8
80GB	70B FP8, 70B INT4 with huge context

2. Memory Bandwidth

Bandwidth determines how fast tokens are generated (inference speed).

GPU	Bandwidth	Tokens/sec (70B INT4)
RTX 4060	272 GB/s	~25 t/s
RTX 4070 Ti	504 GB/s	~50 t/s
RTX 4080 Super	736 GB/s	~80 t/s
RTX 4090	1,008 GB/s	~120 t/s
RTX 5090	1,800 GB/s	~210 t/s
A100 80GB	2,039 GB/s	~250 t/s
H100 80GB	3,350 GB/s	~580 t/s

3. Tensor Cores

All modern NVIDIA GPUs have Tensor Cores that accelerate matrix operations — the fundamental computation in neural networks.

Architecture	Tensor Core Gen	FP8	FP16	BF16	INT8	INT4
Ampere (RTX 30, A100)	3rd Gen	❌	✅	✅	✅	❌
Ada Lovelace (RTX 40)	4th Gen	✅	✅	✅	✅	✅
Blackwell (RTX 50)	5th Gen	✅ (2x speed)	✅	✅	✅	✅
Hopper (H100)	4th Gen	✅	✅	✅	✅	✅

4. Compute Performance (TFLOPS)

Higher TFLOPS means faster training. For inference, bandwidth matters more than compute.

5. Interconnect (for Multi-GPU)

If you plan to use multiple GPUs, interconnects matter:

• NVLink (data center): 600-900 GB/s, critical for model-parallel training
• PCIe 4.0/5.0: 32-64 GB/s, sufficient for data-parallel inference
• No interconnect (software): Works for independent workloads

VRAM Requirements by Model Size

Photo by Shahadat Rahman on Unsplash

Inference (Single Request)

Model Size	FP16	INT8	INT4	INT4 + KV Cache (4K ctx)
1B	2 GB	1 GB	0.5 GB	1 GB
3B	6 GB	3 GB	1.5 GB	3 GB
7B	14 GB	7 GB	3.5 GB	7 GB
8B	16 GB	8 GB	4 GB	8 GB
13B	26 GB	13 GB	6.5 GB	11 GB
34B	68 GB	34 GB	17 GB	26 GB
70B	140 GB	70 GB	35 GB	52 GB
120B	240 GB	120 GB	60 GB	85 GB

Fine-Tuning (QLoRA)

Model Size	AdamW + Gradients	QLoRA Memory	Minimum GPU
7B	48 GB	8-10 GB	RTX 3060 12GB
8B	56 GB	10-14 GB	RTX 4090 24GB
13B	90 GB	14-18 GB	RTX 4090 24GB
34B	240 GB	20-24 GB	RTX 5090 32GB
70B	490 GB	24-32 GB	RTX 5090 / A6000

Tensor Cores and Precision Support

When Each Precision Matters

Precision	Best For	Speed vs FP16	Quality
FP32	Training (precise but slow)	1x (baseline)	Highest
FP16 / BF16	Training, inference	2-8x faster	Near FP32
FP8	Training, inference (H100 gen+)	4-16x faster	Slight degradation
INT8	Inference only	8-16x faster	Minor quality loss
INT4	Inference only	16-32x faster	Noticeable but usable

"FP8 training is the real game-changer of 2026. Combined with a Blackwell GPU, you can train models 2x faster than FP16 with negligible quality loss." — Tim Dettmers, QLoRA creator

Consumer vs Pro vs Datacenter GPUs

Aspect	Consumer (RTX)	Pro (RTX Ada/6000)	Datacenter (H100)
VRAM	12-32 GB	48-96 GB	80 GB
Price	$300-$2,000	$5,000-$10,000	$25,000+
Reliability for 24/7	Moderate	High	Very high
ECC memory	❌	✅	✅
Support	Consumer	Pro support	Enterprise SLA
NVLink	❌	❌ (some)	✅
Best for	Dev, small-scale	70B inference, fine-tuning	Production training

Watch out: Consumer GPUs lose warranty if used in datacenters (NVIDIA enforces this). For office/homelab use, they're fine.

GPU Recommendations by Budget

Budget: Under $500

GPU	Used Price	VRAM	AI Viability
RTX 3060 12GB	$150-200	12GB	Good for 3-7B INT4 models
RTX 3070	$200-280	8GB	Limited (8GB VRAM bottleneck)
Used RTX 2080 Ti	$200-300	11GB	Acceptable for older models

Best pick: RTX 3060 12GB. The 12GB VRAM makes it surprisingly capable for 7B INT4 models.

Mid-Range: $500-$1,000

GPU	Price	VRAM	AI Viability
Used RTX 3090	$700-850	24GB	Excellent — best value AI card
RTX 4070 Ti Super	$799 new	16GB	Good, but less VRAM than 3090
RX 7900 GRE	$549 new	16GB	OK (ROCm support improved)

Best pick: Used RTX 3090. The 24GB VRAM is unmatched at this price point.

High-End: $1,000-$3,000

GPU	Price	VRAM	AI Viability
RTX 5090	$1,999 new	32GB	Excellent — 70B INT4 fits
RTX 4090	$1,599 new	24GB	Very good — 24GB limits 70B
Used RTX A6000	$2,500-3,000	48GB	Excellent for 70B

Best pick: RTX 5090. 32GB VRAM + 2nd-gen FP8 is the sweet spot.

Pro: $3,000+

GPU	Price	VRAM	AI Viability
RTX 6000 Ada (new)	$6,800	48GB	Production-grade
H100 (cloud)	~$2/hr	80GB	Enterprise production
A100 80GB (cloud)	~$1.30/hr	80GB	Cost-effective production

Best pick: Cloud H100 for production, RTX 5090 or used A6000 for local work.

Local vs Cloud GPU

When to Buy Local

• You do heavy daily development/inference (>4 hrs/day)
• Your models fit in 24-32GB VRAM
• You need low latency (local inference)
• Your data privacy requirements prevent cloud usage
• You have a dedicated workspace (noise/power considerations)

When to Use Cloud

• You need 80GB+ VRAM for 70B+ models
• Your workload is bursty (intense training for a few hours occasionally)
• You don't have $2,000+ upfront for hardware
• You need multi-GPU clusters with NVLink
• You want to scale to production without buying hardware

Best strategy: Buy an RTX 5090 for daily work and supplement with cloud H100 instances for large training runs. This is what most AI teams do in 2026. Our cloud GPU comparison covers the best providers.

Multi-GPU Considerations

Setup	VRAM	Use Case	Speedup vs Single
2x RTX 3090	48 GB	70B INT4 inference	~1.8x
2x RTX 4090	48 GB	70B INT4 + context	~1.9x
2x RTX 5090	64 GB	70B FP8	~1.9x
4x RTX 3090	96 GB	70B INT4 training	~2.5-3x
8x H100 (cloud)	640 GB	70B FP8 training	~7x

Important: Multi-GPU inference scales better than training over PCIe. For inference, splitting a model across GPUs is straightforward with vLLM/TensorRT-LLM. For training, NVLink makes a significant difference.

Related Reads

• DePIN + AI: Decentralized Physical Infrastructure Networks for Machine Learning
• Llama 3.1 70B Hardware Requirements: GPU, VRAM, RAM Guide
• Best GPU for Local LLM in 2026 (RTX 4090, 5090, Used Options)

Precision Tradeoffs: When to Use FP8, INT4, and Beyond

FP8 training has become the default for Blackwell GPUs in 2026, offering a 2x speedup over FP16 with negligible quality loss for most models. However, its effectiveness depends on hardware support—only RTX 50-series and Hopper/Blackwell GPUs (H100, RTX 5090) include dedicated FP8 Tensor Cores. For older GPUs like the RTX 3090, FP16 remains the fastest viable option for training, while INT8/INT4 are reserved for inference. When evaluating precision:

• FP8: Ideal for training 7B+ models on supported hardware (e.g., RTX 5090, H100). Use PyTorch’s torch.float8 or NVIDIA’s Transformer Engine for automatic mixed precision.
• INT4: Best for inference on 70B+ models where VRAM is constrained. Tools like bitsandbytes or AWQ quantize models with minimal accuracy loss, but avoid for training.
• BF16: Fallback for GPUs without FP8 support (e.g., RTX 4090). Matches FP16 speed with better numerical stability for large models.

For production deployments, benchmark your model’s sensitivity to precision. Some architectures (e.g., vision transformers) tolerate INT4 better than LLMs, while others (e.g., diffusion models) require FP16/BF16 to avoid artifacts. Always validate with your specific workload—tools like lm-evaluation-harness can quantify precision-induced degradation.

Power and Thermal Considerations for AI Workloads

Consumer GPUs like the RTX 5090 draw 450W under load, requiring a 1000W+ PSU and robust cooling to sustain performance. For 24/7 inference or training, thermal throttling can reduce effective TFLOPS by 10-20% if case airflow is inadequate. Key considerations:

• PSU: Use a Platinum-rated unit with 20-30% headroom (e.g., 1200W for dual RTX 5090s). Cheaper PSUs may fail under sustained high loads.
• Cooling: Liquid cooling (AIO or custom loop) is ideal for overclocked GPUs, but high-end air coolers (e.g., Noctua NH-D15) suffice for stock clocks. Monitor junction temperatures—NVIDIA GPUs throttle at 88°C.
• Power Delivery: Ensure your motherboard has PCIe 5.0 x16 slots with sufficient power phases (e.g., 16+ for RTX 5090). Some boards require 12VHPWR adapters for full power draw.

For multi-GPU setups, spacing cards with PCIe risers or using blower-style GPUs (e.g., RTX 6000 Ada) improves airflow. Datacenter GPUs like the H100 are designed for 300W+ loads in server racks, but their power efficiency (TFLOPS/W) often justifies the cost for large-scale training. Always calculate total system power draw—dual RTX 5090s can exceed 1000W, requiring 240V outlets in some regions.

Software Stack: CUDA, ROCm, and Alternatives in 2026

NVIDIA’s CUDA remains the gold standard for AI workloads, but AMD’s ROCm and emerging alternatives are gaining traction. Here’s how to navigate the ecosystem:

• CUDA: Still the most stable choice for PyTorch/TensorFlow. Use CUDA 12.4+ for Blackwell GPUs (RTX 50-series) to unlock FP8 and 5th-gen Tensor Cores. For older GPUs, CUDA 11.8 is the last version supporting Ampere (RTX 30-series).
• ROCm: AMD’s open-source alternative now supports PyTorch 2.2+ and RX 7000 GPUs, but lacks optimizations for some ops (e.g., flash attention). Best for inference on 7900 XTX/XT GPUs; training is possible but requires manual tuning.
• DirectML: Microsoft’s API for cross-vendor acceleration works on NVIDIA/AMD/Intel GPUs but is limited to Windows and lacks PyTorch integration. Useful for edge deployments with ONNX models.
• OpenCL/Vulkan: Niche options for non-CUDA hardware. Performance lags behind CUDA/ROCm, but tools like cltorch enable basic inference on older GPUs.

For local development, use Docker containers with pre-configured CUDA (e.g., nvcr.io/nvidia/pytorch:23.10-py3) to avoid dependency conflicts. For ROCm, AMD’s official Docker images (rocm/pytorch) simplify setup. Always check framework compatibility—PyTorch 2.3+ is required for FP8 training, while TensorFlow 2.15+ supports Blackwell’s DPX instructions. If you’re using custom kernels (e.g., Triton), ensure they’re compiled for your GPU’s architecture (e.g., sm_90 for RTX 50-series).

Key Takeaways

• VRAM is the single most critical spec for AI/ML GPUs—prioritize 24GB+ (e.g., RTX 3090/5090) to run 7B-70B models locally; 16GB is the bare minimum for 7B INT4 with context.
• Tensor Cores are non-negotiable for deep learning—only NVIDIA RTX 20-series+ (Ampere/Blackwell) or AMD RX 7000+ GPUs have them; avoid GTX cards entirely for AI workloads.
• Memory bandwidth dictates inference speed: RTX 4090 (1,008 GB/s) delivers ~120 tokens/sec for 70B INT4, while RTX 5090 (1,800 GB/s) nearly doubles that (~210 t/s).
• For fine-tuning, QLoRA reduces VRAM needs but still requires 24GB+ for 13B models (e.g., RTX 4090) and 48GB+ for 70B (RTX 5090/A6000); plan for 2-3x model size in VRAM.
• Multi-GPU setups scale poorly for training without NVLink (PCIe 5.0 bottlenecks at ~3x speedup for 4x GPUs) but work well for inference (vLLM/TensorRT-LLM splits models efficiently).
• Hybrid local/cloud strategy is optimal: Use an RTX 5090 for daily development/inference and cloud H100 instances for large-scale training or 70B+ models to balance cost and flexibility.