GPUs

Memory Requirements 11.09 GB

Requires 1 GPUs (based on memory capacity)

10.5 GB

All model weights

0.19 GB

Conversation history cache

0.3 GB

Expert model optimization

0.1 GB

Temporary computation cache

Performance Metrics
Visualize 1558 tok/s →
Total Throughput
1558 tokens/s
Throughput Per User
1558 tokens/s/user
Estimated Latency
96 ms
Max QPS
10.4 queries/s

Estimated using memory bandwidth and tensor compute throughput across 1x NVIDIA H100 (80GB).

Quick Deploy CLI Command
vllm serve openai/gpt-oss-20b --max-model-len 8192
Flags reflect your selected precision, context length, and tensor parallel GPU count.Open GPT-OSS-20B spec page →

Scenario Examples (GPU + Model + Concurrency):

Click these examples to quickly configure popular model deployment scenarios!

📋 Calculation Formula FAQ

How is GPU throughput calculated?
We use a dual-constraint approach: Compute-bound throughput = (GPU TFLOPS × Precision multiplier × 1000) ÷ (2 × Active parameters in billions), and Memory-bound throughput = Memory bandwidth (GB/s) ÷ Total memory access per token (GB). The actual throughput is the minimum of these two values, representing the actual bottleneck.
Why do we consider memory bandwidth limitations?
Based on A100 40GB benchmark data, all tested models were limited by memory bandwidth rather than compute power. This validates our memory bandwidth-based calculation approach, which is more accurate than pure compute-based estimates.
How are MoE models handled differently?
For MoE (Mixture of Experts) models like DeepSeek-R1 (671B total, 37B active), we only consider the active parameters for bandwidth calculations. This means the memory access per token is based on 37B parameters, not the full 671B, making MoE models much more efficient for inference.
What does the system efficiency factor represent?
The system efficiency factor (40% base) accounts for real-world overhead including: framework overhead (Ollama, vLLM), memory access patterns, batch processing efficiency, and hardware utilization. This factor was calibrated using actual A100 40GB benchmark data with an observed average efficiency of 37.2%.
How does L2 cache hit rate affect GPU throughput calculations?
L2 cache significantly impacts memory bandwidth utilization. When model weights are cached in L2, they can be accessed at ~12TB/s instead of ~3.35TB/s from HBM. We use a simplified probability model to calculate cache hit rates: Hit rate = (N-1)/N, where N is the number of concurrent users. This reflects the probability of weight access overlap: 1 user: 0% hit rate (no cache reuse), 2 users: 50% hit rate (second user can reuse weights loaded by first user), 4 users: 75% hit rate, 10 users: 90% hit rate. This approach is based on probability theory: when multiple users access the model simultaneously, the probability that subsequent users access the same weights increases, improving cache efficiency.
How does precision affect performance?
Lower precision formats reduce both memory usage and increase throughput: FP16 (2 bytes/param): Baseline performance, FP8 (1 byte/param): ~2x memory reduction, INT4 (0.5 bytes/param): ~4x memory reduction with 20% performance boost due to better cache utilization. However, aggressive quantization may impact model accuracy, requiring techniques like GPTQ or AWQ to maintain quality.
Why might my results differ from other calculators?
Our calculator uses memory bandwidth as the primary constraint and real benchmark data for calibration, while many others use only compute-based estimates. We also properly handle MoE models and include realistic system efficiency factors, leading to more accurate real-world predictions.
What are the budget-friendly GPU options for LLM deployment?
💰 Budget-Friendly Options (Under $10k): RTX 4090 (24GB) - Best for 7B-13B models, single card setup; RTX 3090 (24GB) - Good value for smaller models and experimentation; Multiple RTX 4060 Ti (16GB) - Cost-effective for distributed inference.
What are the enterprise-grade GPU solutions?
🏢 Enterprise Solutions ($50k+): NVIDIA H100 (80GB) - Industry standard for production LLM deployment; NVIDIA A100 (80GB) - Proven reliability, good for 70B+ models; AMD MI355X (288GB) - Latest CDNA 4, highest memory capacity for massive models; AMD MI350X (288GB) - CDNA 4 air-cooled, excellent price/performance; AMD MI325X (256GB) - CDNA 3 with HBM3E, great for large model deployment; AMD MI300X (192GB) - Proven CDNA 3, excellent for 400B+ models.
What are the GPU requirements for popular AI models?
🆕 Qwen2.5 & Qwen3: Qwen2.5-72B needs 2x H100 with FP8, while Qwen3-235B-A22B (MoE) needs 4x H100. DeepSeek R1: 671B parameters require ~10x H100 with FP8 for optimal performance. Llama 3.1 70B: 2x A100/H100 with FP16, or 7x RTX 4090 for consumer hardware. Llama 3.1 405B: 6x H100 with FP8, or 11x A100 with FP16.
What are the pro tips for GPU optimization?
⚡ Pro Tips: Use FP8/INT8 - Reduce memory usage by 50-75% with minimal quality loss; Consider MoE Models - Qwen3-235B-A22B offers flagship performance with 4x H100 (vs 10x for DeepSeek-R1); Model Parallelism - Split large models across multiple GPUs; Mixed Precision - Combine FP16 inference with FP32 gradients for training; Memory Mapping - Use CPU RAM for model storage, GPU for active layers.