--- title: "How to choose a GPU for LLM inference | UZ Dynamics" description: "Match a GPU to your model, context and concurrent requests. An 8B memory example, quantization limits and a practical latency test plan." url: "https://uzdynamics.ai/en/guides/llm-inference" lang: "en" image: "https://uzdynamics.ai/en/opengraph-image/card?1237d0f845bd82b9" alternates: ru: "https://uzdynamics.ai/ru/guides/llm-inference" uz: "https://uzdynamics.ai/uz/guides/llm-inference" en: "https://uzdynamics.ai/en/guides/llm-inference" --- > For an index of this site's pages, see [llms.txt](https://uzdynamics.ai/llms.txt). A markdown copy of each page is at its address with `.md` appended. # How to choose a GPU for LLM inference Describe the model and traffic first, then check memory and measure latency. VRAM capacity can rule out unsuitable configurations, but it cannot predict response speed on its own. Author: [UZ Dynamics](https://uzdynamics.ai/en#company) Sources checked: 9 October 2026 Updated: 9 October 2026 Published: 9 October 2026 ## Start with the response requirements Record the exact checkpoint and revision, engine and library versions, weight format, and maximum input and output lengths. Context includes both: a 7,000-token input with up to 1,000 output tokens needs an 8,000-token budget, not 7,000. Four responses being generated at once are a different workload from four registered users. For chat, measure the wait for the first token and gaps between tokens. For batch processing, measure how much work finishes within the deadline. - Short chat: check latency at ordinary and peak concurrency. - RAG with long documents: test long inputs and KV cache usage separately. - Batch processing: specify the deadline and whether requests can wait in a queue. ## Allow memory beyond the weights Worked example: exactly 8 billion parameters stored in BF16 need 16 billion bytes, or about 14.9 GiB, for weights alone. Add the KV cache, temporary tensors and engine buffers. This arithmetic does not establish that every checkpoint labelled 8B will run on a particular GPU. Ideal 4-bit packing would use 4 billion bytes for the weights, but real formats include quantization scales and may keep some layers at higher precision. Check engine and GPU support for the format, quality on your own examples, and actual memory after loading. Weight bytes ≈ parameter count × bits per parameter / 8 Section sources: - [Hugging Face Transformers: Bitsandbytes](https://huggingface.co/docs/transformers/en/quantization/bitsandbytes) ## Compare configurations on the same workload Use the same checkpoint, inputs, output lengths, engine versions and settings. Separate cold starts, warm runs and tests that reuse a cached prefix. Equal GPU counts do not imply equal speed: memory, connections between GPUs and supported operations differ. vLLM exposes metrics for time to first token, inter-token latency, queue time and token counts. Configuration selection needs distributions such as p50 and p95, rather than only the average speed of one request. - Record peak VRAM, out-of-memory errors and the proportion of successful requests. - Measure TTFT and inter-token latency at 1, 4 and your target peak concurrency. - Repeat with typical and long inputs, saving the launch configuration. - Once billing terms are known, compare total job cost, including idle time, storage and data transfer. Section sources: - [vLLM: Production Metrics](https://docs.vllm.ai/en/stable/usage/metrics/) ## What to provide for configuration planning The site calculator gives an initial estimate from open formulas. To choose a final configuration, provide the model name, weight format, input and output lengths, concurrency, target latency, run duration and data constraints. Secrets and user data are not needed for this. This guide contains no UZ Dynamics measurements for these scenarios. A test on an agreed configuration must establish the results before anyone promises tokens per second, a user count or a cost per request. ## GPUs to compare Compare L4, L40S, A100 and H100 specifications after calculating memory. The model pages describe hardware specifications; they do not confirm availability or recommend a specific SKU without testing. - [L4](https://uzdynamics.ai/en/gpus/l4) - [L40S](https://uzdynamics.ai/en/gpus/l40s) - [A100](https://uzdynamics.ai/en/gpus/a100) - [H100](https://uzdynamics.ai/en/gpus/h100) ## Estimate your workload and request access UZ Dynamics is in early access. These guides help describe a workload; they do not confirm GPU availability, a ready environment or execution speed. Availability, the exact configuration, prices and access terms need confirmation when discussing your request. [Open the GPU calculator](https://uzdynamics.ai/en/sizing) [All GPU models](https://uzdynamics.ai/en/gpus) [All guides](https://uzdynamics.ai/en/guides) [Request early access](https://uzdynamics.ai/en/request?source=guide%3Allm-inference) ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"UZ Dynamics","item":"https://uzdynamics.ai/en"},{"@type":"ListItem","position":2,"name":"GPU planning guides","item":"https://uzdynamics.ai/en/guides"},{"@type":"ListItem","position":3,"name":"How to choose a GPU for LLM inference","item":"https://uzdynamics.ai/en/guides/llm-inference"}]} {"@context":"https://schema.org","@type":"Article","@id":"https://uzdynamics.ai/en/guides/llm-inference#article","headline":"How to choose a GPU for LLM inference","description":"Match a GPU to your model, context and concurrent requests. An 8B memory example, quantization limits and a practical latency test plan.","inLanguage":"en","datePublished":"2026-10-09","dateModified":"2026-10-09","author":{"@type":"Organization","@id":"https://uzdynamics.ai/#organization","name":"UZ Dynamics","url":"https://uzdynamics.ai/"},"publisher":{"@type":"Organization","@id":"https://uzdynamics.ai/#organization","name":"UZ Dynamics","url":"https://uzdynamics.ai/"},"mainEntityOfPage":{"@type":"WebPage","@id":"https://uzdynamics.ai/en/guides/llm-inference"},"url":"https://uzdynamics.ai/en/guides/llm-inference"} ```