What LLMs Can NVIDIA H20 (96GB) Run?
Hardware capability guide for NVIDIA H20: 96GB memory with 4000 GB/s bandwidth across single-card and multi-GPU deployments.
NVIDIA H20 Hardware Specifications
VRAM / Card
96 GB
Mem Bandwidth
4000 GB/s
FP16 Compute
148 TFLOPS
Architecture
Hopper
Release Year
2024
Available VRAM
96 GB
Total Bandwidth
4,000 GB/s
Fits FP16 / FP8
45
Fits w/ INT4
61 / 85
DeepSeek-R1-Distill-Qwen-32B
32B • DeepSeek
FP16
68.27 GB
FP8
36 GB
INT4
19.86 GB
~46 tok/s (FP16)Open VRAM Calculator
DeepSeek-R1-Distill-Qwen-14B
14B • DeepSeek
FP16
32.27 GB
FP8
18 GB
INT4
10.86 GB
~46 tok/s (FP16)Open VRAM Calculator
DeepSeek-R1-Distill-Qwen-7B
7B • DeepSeek
FP16
18.27 GB
FP8
11 GB
INT4
7.36 GB
~46 tok/s (FP16)Open VRAM Calculator
DeepSeek-R1-Distill-Qwen-1.5B
1.5B • DeepSeek
FP16
7.27 GB
FP8
5.5 GB
INT4
4.61 GB
~46 tok/s (FP16)Open VRAM Calculator
DeepSeek-R1-Distill-Llama-70B
70B • DeepSeek
FP16
144.27 GB
FP8
74 GB
INT4
38.86 GB
~92 tok/s (FP8)Open VRAM Calculator
DeepSeek-R1-Distill-Llama-8B
8B • DeepSeek
FP16
20.27 GB
FP8
12 GB
INT4
7.86 GB
~46 tok/s (FP16)Open VRAM Calculator
Llama 4 ScoutMoE 17B
109B • Llama 4
FP16
220.71 GB
FP8
111.21 GB
INT4
56.46 GB
~400 tok/s (INT4)Open VRAM Calculator
Llama 3 8B
8B • Llama 3
FP16
17.81 GB
FP8
9.31 GB
INT4
5.06 GB
~213 tok/s (FP16)Open VRAM Calculator
Llama 3 70B
70B • Llama 3
FP16
149.56 GB
FP8
78.31 GB
INT4
42.68 GB
~49 tok/s (FP8)Open VRAM Calculator
Llama 3.1 8B
8B • Llama 3
FP16
17.81 GB
FP8
9.31 GB
INT4
5.06 GB
~213 tok/s (FP16)Open VRAM Calculator
Llama 3.1 70B
70B • Llama 3
FP16
149.56 GB
FP8
78.31 GB
INT4
42.68 GB
~49 tok/s (FP8)Open VRAM Calculator
Llama 3.2 1B
1B • Llama 3
FP16
2.55 GB
FP8
1.43 GB
INT4
0.86 GB
~1700 tok/s (FP16)Open VRAM Calculator
Llama 3.2 3B
3B • Llama 3
FP16
7.23 GB
FP8
3.79 GB
INT4
2.07 GB
~567 tok/s (FP16)Open VRAM Calculator
Llama 3.2 11B
11B • Llama 3
FP16
24.11 GB
FP8
12.61 GB
INT4
6.86 GB
~155 tok/s (FP16)Open VRAM Calculator
Llama 3.2 90B
90B • Llama 3
FP16
191.57 GB
FP8
100.32 GB
INT4
54.7 GB
~76 tok/s (INT4)Open VRAM Calculator
Qwen3.5-122B-A10BMoE 10B
122B • Qwen 3.5
FP16
246.01 GB
FP8
123.51 GB
INT4
62.26 GB
~680 tok/s (INT4)Open VRAM Calculator
Qwen3.5-35B-A3BMoE 3B
35B • Qwen 3.5
FP16
70.98 GB
FP8
35.66 GB
INT4
18.01 GB
~567 tok/s (FP16)Open VRAM Calculator
Qwen3.5-27B
27B • Qwen 3.5
FP16
59.22 GB
FP8
30.97 GB
INT4
16.85 GB
~63 tok/s (FP16)Open VRAM Calculator
Qwen3-Next-80B-A3B-InstructMoE 3B
81B • Qwen 3-Next
FP16
163.48 GB
FP8
81.91 GB
INT4
41.13 GB
~1133 tok/s (FP8)Open VRAM Calculator
Qwen3-Next-80B-A3B-ThinkingMoE 3B
81B • Qwen 3-Next
FP16
163.48 GB
FP8
81.91 GB
INT4
41.13 GB
~1133 tok/s (FP8)Open VRAM Calculator
Qwen2.5-Math-72B
72B • Qwen 2
FP16
153.76 GB
FP8
80.51 GB
INT4
43.88 GB
~47 tok/s (FP8)Open VRAM Calculator
Qwen2.5-72B
72B • Qwen 2
FP16
153.76 GB
FP8
80.51 GB
INT4
43.88 GB
~47 tok/s (FP8)Open VRAM Calculator
Qwen2-72B
72B • Qwen 2
FP16
153.76 GB
FP8
80.51 GB
INT4
43.88 GB
~47 tok/s (FP8)Open VRAM Calculator
Qwen QwQ-32B
32B • Qwen QwQ
FP16
69.1 GB
FP8
36.16 GB
INT4
19.69 GB
~53 tok/s (FP16)Open VRAM Calculator
Qwen2.5-Coder-32B
32B • Qwen 2
FP16
69.1 GB
FP8
36.16 GB
INT4
19.69 GB
~53 tok/s (FP16)Open VRAM Calculator
Qwen3-32B
32B • Qwen 3
FP16
69.23 GB
FP8
36.23 GB
INT4
19.73 GB
~53 tok/s (FP16)Open VRAM Calculator
Qwen2.5-32B
32B • Qwen 2
FP16
69.1 GB
FP8
36.16 GB
INT4
19.69 GB
~53 tok/s (FP16)Open VRAM Calculator
Qwen2-32B
32B • Qwen 2
FP16
69.1 GB
FP8
36.16 GB
INT4
19.69 GB
~53 tok/s (FP16)Open VRAM Calculator
Qwen3-30B-A3BMoE 3B
30B • Qwen 3
FP16
61.1 GB
FP8
30.73 GB
INT4
15.54 GB
~567 tok/s (FP16)Open VRAM Calculator
Qwen2.5-14B
14B • Qwen 2
FP16
30.66 GB
FP8
16.04 GB
INT4
8.72 GB
~121 tok/s (FP16)Open VRAM Calculator
Qwen3-14B
14B • Qwen 3
FP16
30.66 GB
FP8
16.04 GB
INT4
8.72 GB
~121 tok/s (FP16)Open VRAM Calculator
Qwen2-14B
14B • Qwen 2
FP16
30.66 GB
FP8
16.04 GB
INT4
8.72 GB
~121 tok/s (FP16)Open VRAM Calculator
Qwen2.5-7B
7B • Qwen 2
FP16
15.21 GB
FP8
7.96 GB
INT4
4.33 GB
~243 tok/s (FP16)Open VRAM Calculator
Qwen2.5-3B
3B • Qwen 2
FP16
6.57 GB
FP8
3.46 GB
INT4
1.91 GB
~567 tok/s (FP16)Open VRAM Calculator
Qwen2.5-1.5B
1.5B • Qwen 2
FP16
3.49 GB
FP8
1.89 GB
INT4
1.1 GB
~1133 tok/s (FP16)Open VRAM Calculator
Qwen2.5-0.5B
0.5B • Qwen 2
FP16
1.36 GB
FP8
0.83 GB
INT4
0.57 GB
~3400 tok/s (FP16)Open VRAM Calculator
Qwen3-0.6B
0.6B • Qwen 3
FP16
1.94 GB
FP8
1.12 GB
INT4
0.71 GB
~2833 tok/s (FP16)Open VRAM Calculator
Qwen3-1.7B
1.7B • Qwen 3
FP16
4.14 GB
FP8
2.22 GB
INT4
1.26 GB
~1000 tok/s (FP16)Open VRAM Calculator
Mistral-Large-Instruct-2407
123B • Mistral
FP16
261.15 GB
FP8
136.77 GB
INT4
74.59 GB
~55 tok/s (INT4)Open VRAM Calculator
Mistral-Nemo-12B
12B • Mistral
FP16
26.46 GB
FP8
13.83 GB
INT4
7.52 GB
~142 tok/s (FP16)Open VRAM Calculator
Mistral-7B-v0.3
7B • Mistral
FP16
15.71 GB
FP8
8.21 GB
INT4
4.46 GB
~243 tok/s (FP16)Open VRAM Calculator
Mixtral-8x7BMoE 12.9B
46.7B • Mixtral
FP16
95.7 GB
FP8
48.5 GB
INT4
24.9 GB
~132 tok/s (FP16)Open VRAM Calculator
Mixtral-8x22BMoE 39B
141B • Mixtral
FP16
287.68 GB
FP8
145.81 GB
INT4
74.87 GB
~174 tok/s (INT4)Open VRAM Calculator
Gemma-2-27B
27B • Gemma
FP16
59.6 GB
FP8
31.16 GB
INT4
16.94 GB
~63 tok/s (FP16)Open VRAM Calculator
Gemma-2-9B
9B • Gemma
FP16
21.53 GB
FP8
11.22 GB
INT4
6.06 GB
~189 tok/s (FP16)Open VRAM Calculator
Phi-3.5-MoEMoE 6.6B
42B • Phi
FP16
85.67 GB
FP8
43.17 GB
INT4
21.92 GB
~258 tok/s (FP16)Open VRAM Calculator
Phi-3-3.8B
3.8B • Phi
FP16
11.02 GB
FP8
5.72 GB
INT4
3.07 GB
~447 tok/s (FP16)Open VRAM Calculator
CodeLlama-34B
34B • CodeLlama
FP16
72.93 GB
FP8
38.18 GB
INT4
20.8 GB
~50 tok/s (FP16)Open VRAM Calculator
CodeLlama-13B
13B • CodeLlama
FP16
33.56 GB
FP8
17.44 GB
INT4
9.37 GB
~131 tok/s (FP16)Open VRAM Calculator
CodeLlama-7B
7B • CodeLlama
FP16
18.71 GB
FP8
9.71 GB
INT4
5.21 GB
~243 tok/s (FP16)Open VRAM Calculator
GLM-4.5-AirMoE 12B
106B • GLM-4
FP16
217.73 GB
FP8
110.48 GB
INT4
56.85 GB
~213 tok/s (INT4)Open VRAM Calculator
GLM-4.5-Air-BaseMoE 12B
106B • GLM-4
FP16
217.73 GB
FP8
110.48 GB
INT4
56.85 GB
~213 tok/s (INT4)Open VRAM Calculator
GLM-4.7-FlashMoE 3B
30B • GLM-4
FP16
66.1 GB
FP8
34.66 GB
INT4
18.94 GB
~53 tok/s (FP16)Open VRAM Calculator
GPT-OSS-120BMoE 5.1B
117B • GPT-OSS
FP16
235.65 GB
FP8
118.09 GB
INT4
59.31 GB
~1333 tok/s (INT4)Open VRAM Calculator
GPT-OSS-20BMoE 3.6B
21B • GPT-OSS
FP16
43.15 GB
FP8
21.78 GB
INT4
11.09 GB
~472 tok/s (FP16)Open VRAM Calculator
NVIDIA H20 1x / 2x / 4x Multi-GPU LLM Compatibility Matrix
Precomputed at 8K context, batch size 1 across key benchmark models.
| LLM Model | FP8 VRAM Req | 1× (96GB) | 2× (192GB) | 4× (384GB) |
|---|---|---|---|---|
| DeepSeek-R1(671B) | 675 GB | Needs 4× | Needs 4× | INT4 |
| DeepSeek-R1-Distill-Qwen-32B(32B) | 36 GB | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 |
| Llama 4 Scout(109B) | 111.21 GB | INT4 | FP8 / INT4 | FP16 / FP8 / INT4 |
| Llama 3.1 8B(8B) | 9.31 GB | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 |
| Llama 3.1 70B(70B) | 78.31 GB | FP8 / INT4 | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 |
| Qwen3-235B-A22B(235B) | 237.95 GB | Needs 2× | INT4 | FP8 / INT4 |
| Qwen3-32B(32B) | 36.23 GB | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 |
| Qwen3-14B(14B) | 16.04 GB | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 |
| Mixtral-8x7B(46.7B) | 48.5 GB | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 |
| GPT-OSS-20B(21B) | 21.78 GB | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 | FP16 / FP8 / INT4 |
Dedicated Compatibility Pages for All 22 GPUs
NVIDIA RTX 5090 (32GB)NVIDIA RTX 5080 (16GB)NVIDIA RTX 5070 Ti (16GB)NVIDIA RTX 4090 (24GB)NVIDIA RTX 4080 SUPER (16GB)NVIDIA RTX 4070 Ti Super (16GB)NVIDIA RTX 4060 Ti (16GB)NVIDIA RTX 3090 (24GB)AMD Radeon RX 7900 XTX (24GB)NVIDIA RTX PRO 6000 Blackwell (96GB)Apple M4 Max (128GB)Apple M4 Max (64GB)Apple M4 Pro (48GB)Apple M3 Ultra (192GB)NVIDIA B200 (192GB)NVIDIA H200 (141GB)NVIDIA H100 (80GB)NVIDIA H20 (96GB)NVIDIA A100 (80GB)NVIDIA L40S (48GB)AMD Instinct MI300X (192GB)Huawei Ascend 910B (64GB)
NVIDIA H20 Local LLM Deployment FAQ
tools.gpuCompatibility.gpuDetail.faq.q1▾
With 96GB of memory on a single NVIDIA H20, 61 benchmark models in our registry fit cleanly in FP16, FP8, or INT4 at 8K context.
What generation speed can I expect from NVIDIA H20?▾
With 4000 GB/s memory bandwidth and 96GB memory, single-stream decode speed scales inversely with active model weight size read per token.
When should I scale to 2x or 4x NVIDIA H20?▾
Multi-GPU tensor parallelism becomes necessary when model weights plus KV cache exceed 96GB or when serving high-concurrency batches.
Explore Related LLM & GPU Tools
Browse full AI toolkit directory →Popular Model VRAM Calculators
DeepSeek-R1 (671B)DeepSeek-V3 (671B)DeepSeek-V4-Pro (1.6T)DeepSeek-V4-Flash (284B)DeepSeek-R1-Distill-Qwen-32BDeepSeek-R1-Distill-Qwen-14BDeepSeek-R1-Distill-Qwen-7BDeepSeek-R1-Distill-Llama-70BLlama 4 Maverick (400B)Llama 4 Scout (109B)Llama 3.1 405BLlama 3.1 70BLlama 3.1 8BLlama 3.2 3BQwen3.5-397B-A17BQwen3-235B-A22BQwen3-Coder-480B-A35BQwen3-32B
GPU Compatibility Guides
NVIDIA RTX 5090 (32GB)NVIDIA RTX 5080 (16GB)NVIDIA RTX 5070 Ti (16GB)NVIDIA RTX 4090 (24GB)NVIDIA RTX 4080 SUPER (16GB)NVIDIA RTX 4070 Ti Super (16GB)NVIDIA RTX 4060 Ti (16GB)NVIDIA RTX 3090 (24GB)AMD Radeon RX 7900 XTX (24GB)NVIDIA RTX PRO 6000 Blackwell (96GB)Apple M4 Max (128GB)Apple M4 Max (64GB)Apple M4 Pro (48GB)Apple M3 Ultra (192GB)