--- title: "GPU sizing for your workload: early access | UZ Dynamics" description: "Early access by request: GPU sizing by open formulas. Memory, configurations and GPU-hours a month for inference, fine-tuning, training and rendering." url: "https://uzdynamics.ai/en/sizing" lang: "en" image: "https://uzdynamics.ai/en/opengraph-image/card?1237d0f845bd82b9" alternates: ru: "https://uzdynamics.ai/ru/sizing" uz: "https://uzdynamics.ai/uz/sizing" en: "https://uzdynamics.ai/en/sizing" --- > For an index of this site's pages, see [llms.txt](https://uzdynamics.ai/llms.txt). A markdown copy of each page is at its address with `.md` appended. Early-access requests # GPU sizing for your workload An estimate from open formulas and specs from NVIDIA’s official materials. We’ll discuss the final configuration in our reply. We’ll reply with terms for your workload, as we don’t publish prices yet. Nothing to pay before the contract. The sizing tool needs JavaScript. Below is the smallest fit by memory for each workload. ## Result Example. About 87 GB of GPU memory An estimate from open formulas, not a benchmark or an invoice. **Fit in memory** | Configuration | Total VRAM | Memory bandwidth | | --- | --- | --- | | 1 × RTX PRO 6000 bandwidth 1.6 TB/s Selected | 96 GB 96 GB/GPU 9 GB spare | 1.6 TB/s | | 1 × H200 bandwidth 4.8 TB/s | 141 GB 141 GB/GPU 54 GB spare | 4.8 TB/s | | 1 × B200 bandwidth 7.7 TB/s | 180 GB 180 GB/GPU 93 GB spare | 7.7 TB/s | | 1 × B300 bandwidth 7.7 TB/s | 270 GB 270 GB/GPU 183 GB spare | 7.7 TB/s | | 2 × RTX 6000 Ada PCIe 4 bandwidth 0.96 TB/s | 96 GB 48 GB/GPU 9 GB spare | 0.96 TB/s | 720 GPU-hours a month 1 × 24 h × 30 days Each GPU is busy 99% of an average month (730 h) We’ll reply with terms for dedicated servers. [Rent or buy: what makes up the cost](https://uzdynamics.ai/en/sizing#rent-vs-buy) - We’ll confirm availability in our reply. - Access: UZ Dynamics Console, SSH - Suggested format: Dedicated servers and clusters * Memory bandwidth drives generation speed. Headroom covers context and concurrent requests; long context needs more memory. [Request this configuration](https://uzdynamics.ai/en/request?source=sizing&task=infer&sub=llm&p=70&prec=fp8&gpu=rtx-pro-6000&n=1&mode=mo&h=24&d=30&fmt=dedicated&example=1) Not sure about the parameters? Describe the workload in your own words and we’ll reply with the terms. There’s no need to pick a GPU first. [Describe the workload](https://uzdynamics.ai/en/request?source=sizing-help) **Rent or buy GPU servers** Owning servers pays off with steady high utilisation and an in-house ops team. Renting suits uneven load or changing workloads. - ### Upfront cash - **Buy:** A large upfront payment before any work (or leasing/financing) - **Rent:** Payments under a contract, no hardware purchase - ### Time to first job - **Buy:** Delivery, installation and setup - **Rent:** Lead time being confirmed - ### Power and cooling - **Buy:** Your monthly cost - **Rent:** Not your cost: covered by whoever provides the GPUs - ### Operations - **Buy:** Your own engineers for drivers and repairs - **Rent:** Hardware upkeep isn’t on you; who maintains it will be set out in the terms - ### Utilisation - **Buy:** An idle GPU is still paid for - **Rent:** You pay for the time you rent, not years of ownership. GPU count per format comes with the terms - ### New GPU generations - **Buy:** A server stays the generation you bought - **Rent:** The catalogue spans several generations, from Ampere to Blackwell - ### Networking between servers - **Buy:** Your own switches and links - **Rent:** We’ll confirm in our reply ### Your utilisation [Estimate utilisation in the sizing tool](https://uzdynamics.ai/en/sizing#sizing) ### Flagship GPUs by generation 1. 2020: A100 (Ampere) 2. 2022: H100 (Hopper) 3. 2023: H200 (Hopper) 4. 2024: B200 (Blackwell) 5. 2025: B300 (Blackwell) Announcement years per NVIDIA. ## How we estimate Inference: memory = parameters × bytes per parameter × 1.2. Bytes per parameter: BF16 2, FP8 1, INT4 0.5 (the “2 GB per billion parameters in BF16” rule of thumb, [Hugging Face Transformers](https://huggingface.co/docs/transformers/main/en/llm_tutorial_optimization)). The 20% headroom for KV cache and activations is our assumption for moderate context and a few concurrent requests; long context and heavy load need more room for the KV cache. In FP8 and INT4, part of a language model’s weights (embeddings, output layer) stays 16-bit; we take its share from published checkpoints. Example: for a 70B model in 4 bits (2.2B of it in 16 bits) the estimate is about 46 GB, and a real 4-bit checkpoint takes about 40 GB ([AWQ-INT4 on Hugging Face](https://huggingface.co/hugging-quants/Meta-Llama-3.1-70B-Instruct-AWQ-INT4)). LoRA: parameters × 2 bytes × 1.3. QLoRA: weights as for INT4 inference × 1.4, so a 65B model (about 47 GB) fits in 48 GB, as in the [QLoRA paper](https://arxiv.org/abs/2305.14314). Full fine-tuning: 18 bytes per parameter (weights, gradients and AdamW state in mixed precision, [Hugging Face](https://huggingface.co/docs/transformers/model_memory_anatomy)) × 1.2. Training from scratch: memory as for full fine-tuning. That’s the minimum number of GPUs to hold the weights, gradients and optimiser state. GPU count: the smallest of 1, 2, 4 and 8 whose total memory covers the need. This assumes the model is split across the GPUs (tensor or pipeline parallelism); otherwise one GPU needs enough memory. Beyond that we count in multiples of 8, since a server usually holds up to 8 GPUs (an assumption). On L4 and L40S, ECC takes part of the memory by default (NVIDIA data); on the RTX A6000, A5000 and RTX 6000 Ada, when it is switched on. We still count the full memory: NVIDIA gives it in binary gigabytes, and what ECC leaves is at least as many decimal ones. Rendering: for full speed the scene must fit in one GPU’s memory (when it doesn’t, Blender falls back to system memory, which is slower). GPUs usually don’t pool memory; NVIDIA GPUs joined by NVLink are the exception ([Blender Manual](https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html)). The estimate counts one GPU. Only GPUs with RT cores (hardware ray tracing) fit. Video: encoding needs NVENC, analytics needs NVDEC (engine counts from [NVIDIA’s encode/decode matrix](https://developer.nvidia.com/video-encode-decode-support-matrix)). FP64: only GPUs whose FP64 per NVIDIA is at least a quarter of FP32. For matrix-heavy work (DGEMM), FP64 Tensor Core matters more: 67 TFLOPS on H100 SXM and H200 SXM, 37 on B200 (NVIDIA: H100 and H200 pages, Blackwell Technical Brief). So rows are ordered by FP64 Tensor Core. GPU-hours: GPUs × hours a day × days. Utilisation: hours a day × days ÷ 730 (average hours in a month), capped at 100%. At 70% or more we suggest dedicated servers. That is our rule of thumb, not a price calculation: at that load, a dedicated GPU sits idle at most 30% of the time. Row order: fewer GPUs, then less spare memory; on a tie, faster memory first. For multi-GPU training and full fine-tuning, NVLink rows come first. For video, less memory first, then, when encoding, more NVENC engines. For rendering, less spare memory, then memory bandwidth. ## Assumptions - Memory headroom ×1.2 for inference, ×1.2 for full fine-tuning and training from scratch, ×1.3 for LoRA, ×1.4 for QLoRA - Sizing assumption: up to 8 GPUs per server - We use NVIDIA’s usable GPU memory, not the chip’s maximum HBM capacity (B300 270 of 288 GB, B200 180 of 192 GB) - GPU memory is taken in NVIDIA’s (binary) gigabytes and the need in decimal ones, so the estimate keeps a margin of about 7%, and under 1% where ECC takes part of the memory The estimate follows open formulas and NVIDIA’s published specifications: it is not a measurement, not a guarantee of availability and not a price offer. We confirm the final configuration, availability and terms in our reply to your request. ## Smallest fit by memory | Workload | Preset | Memory bandwidth | Request | | --- | --- | --- | --- | | Inference | 8B LLM in BF16 | 0.77 TB/s | [Request 1 × RTX A5000 (8B LLM in BF16)](https://uzdynamics.ai/en/request?source=preset%3Ainfer&task=infer&sub=llm&p=8&prec=bf16&gpu=rtx-a5000&n=1&mode=mo&h=24&d=30&fmt=dedicated) | | Inference | 70B LLM in FP8 | 1.6 TB/s | [Request 1 × RTX PRO 6000 (70B LLM in FP8)](https://uzdynamics.ai/en/request?source=preset%3Ainfer&task=infer&sub=llm&p=70&prec=fp8&gpu=rtx-pro-6000&n=1&mode=mo&h=24&d=30&fmt=dedicated) | | Fine-tuning | LoRA on an 8B model | 0.77 TB/s | [Request 1 × RTX A5000 (LoRA on an 8B model)](https://uzdynamics.ai/en/request?source=preset%3Afinetune&task=finetune&sub=lora&p=8&gpu=rtx-a5000&n=1&mode=once&hours=24&fmt=cloud) | | Fine-tuning | QLoRA on a 70B model | 3.35 TB/s | [Request 1 × H100 (QLoRA on a 70B model)](https://uzdynamics.ai/en/request?source=preset%3Afinetune&task=finetune&sub=qlora&p=70&gpu=h100&n=1&mode=once&hours=24&fmt=cloud) | | Training | 3B LLM from scratch | 3.35 TB/s | [Request 1 × H100 (3B LLM from scratch)](https://uzdynamics.ai/en/request?source=preset%3Atrain&task=train&sub=llmPretrain&p=3&gpu=h100&n=1&mode=once&hours=168&fmt=cloud) | | Rendering and video | 3D scene up to 16 GB | 0.77 TB/s | [Request 1 × RTX A5000 (3D scene up to 16 GB)](https://uzdynamics.ai/en/request?source=preset%3Arender&task=render&sub=render3d&scene=16&gpu=rtx-a5000&n=1&mode=mo&h=8&d=22&fmt=cloud) | | Rendering and video | Video encoding | 0.3 TB/s | [Request 1 × L4 (Video encoding)](https://uzdynamics.ai/en/request?source=preset%3Arender&task=render&sub=encode&gpu=l4&n=1&mode=mo&h=24&d=30&fmt=dedicated) | | Scientific computing (FP64) | Computing on one GPU | 3.35 TB/s | [Request 1 × H100 (Computing on one GPU)](https://uzdynamics.ai/en/request?source=preset%3Afp64&task=fp64&sub=fp64&gpu=h100&n=1&mode=once&hours=72&fmt=cloud) | [More than 8 GPUs? Discuss a cluster](https://uzdynamics.ai/en/enterprise?source=sizing-presets-cluster) [All GPU specifications](https://uzdynamics.ai/en/gpus) [GPU planning guides](https://uzdynamics.ai/en/guides) ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"UZ Dynamics","item":"https://uzdynamics.ai/en"},{"@type":"ListItem","position":2,"name":"GPU sizing for your workload","item":"https://uzdynamics.ai/en/sizing"}]} ```