GPU sizing for your workload

Inference: memory = parameters × bytes per parameter × 1.2. Bytes per parameter: BF16 2, FP8 1, INT4 0.5 (the “2 GB per billion parameters in BF16” rule of thumb, Hugging Face Transformers (opens in a new tab)). The 20% headroom for KV cache and activations is our assumption for moderate context and a few concurrent requests; long context and heavy load need more room for the KV cache. In FP8 and INT4, part of a language model’s weights (embeddings, output layer) stays 16-bit; we take its share from published checkpoints. Example: for a 70B model in 4 bits (2.2B of it in 16 bits) the estimate is about 46 GB, and a real 4-bit checkpoint takes about 40 GB (AWQ-INT4 on Hugging Face (opens in a new tab)).

LoRA: parameters × 2 bytes × 1.3. QLoRA: weights as for INT4 inference × 1.4, so a 65B model (about 47 GB) fits in 48 GB, as in the QLoRA paper (opens in a new tab). Full fine-tuning: 18 bytes per parameter (weights, gradients and AdamW state in mixed precision, Hugging Face (opens in a new tab)) × 1.2.

Training from scratch: memory as for full fine-tuning. That’s the minimum number of GPUs to hold the weights, gradients and optimiser state.

GPU count: the smallest of 1, 2, 4 and 8 whose total memory covers the need. This assumes the model is split across the GPUs (tensor or pipeline parallelism); otherwise one GPU needs enough memory. Beyond that we count in multiples of 8, since a server usually holds up to 8 GPUs (an assumption). On L4 and L40S, ECC takes part of the memory by default (NVIDIA data); on the RTX A6000, A5000 and RTX 6000 Ada, when it is switched on. We still count the full memory: NVIDIA gives it in binary gigabytes, and what ECC leaves is at least as many decimal ones.

Rendering: for full speed the scene must fit in one GPU’s memory (when it doesn’t, Blender falls back to system memory, which is slower). GPUs usually don’t pool memory; NVIDIA GPUs joined by NVLink are the exception (Blender Manual (opens in a new tab)). The estimate counts one GPU. Only GPUs with RT cores (hardware ray tracing) fit. Video: encoding needs NVENC, analytics needs NVDEC (engine counts from NVIDIA’s encode/decode matrix (opens in a new tab)).

FP64: only GPUs whose FP64 per NVIDIA is at least a quarter of FP32. For matrix-heavy work (DGEMM), FP64 Tensor Core matters more: 67 TFLOPS on H100 SXM and H200 SXM, 37 on B200 (NVIDIA: H100 and H200 pages, Blackwell Technical Brief). So rows are ordered by FP64 Tensor Core.

GPU-hours: GPUs × hours a day × days. Utilisation: hours a day × days ÷ 730 (average hours in a month), capped at 100%. At 70% or more we suggest dedicated servers. That is our rule of thumb, not a price calculation: at that load, a dedicated GPU sits idle at most 30% of the time.

Row order: fewer GPUs, then less spare memory; on a tie, faster memory first. For multi-GPU training and full fine-tuning, NVLink rows come first. For video, less memory first, then, when encoding, more NVENC engines. For rendering, less spare memory, then memory bandwidth.