LLM VRAM: weights, KV cache and context

Budget for several memory components

Weights are loaded for model computation. The KV cache stores attention keys and values for tokens already processed. Activations, workspace buffers, CUDA graphs and distributed operations use additional memory whose size depends on the implementation.

The site calculator uses a headroom factor for initial selection. It does not model the KV cache architecture of every checkpoint. Long contexts or high concurrency need a separate estimate and measurement.

VRAM ≈ weights + KV cache + activations and workspace

A formula for a conventional KV cache

For a decoder-only model with full attention and identical layers, estimate capacity from the layer count L, KV head count Hkv, head dimension D, bytes per element b and total retained tokens T across active sequences. The factor of 2 accounts for keys and values.

Read the KV head count from the model configuration. With GQA or MQA it can differ from the query head count. The formula estimates logical storage; block allocation, engine reserves and copies used in distributed execution can change actual usage.

KV bytes ≈ 2 × L × Hkv × D × b × T

Example: 1 GiB becomes 4 GiB

This is an illustrative architecture, not a measured checkpoint: 32 layers, 8 KV heads, head dimension 128, and BF16 cache elements at 2 bytes. That gives 131,072 bytes per token. At 8,192 retained tokens, one sequence needs 1 GiB of KV cache; four independent sequences of that length need 4 GiB.

If all four sequences reach 32,768 tokens, logical KV storage reaches 16 GiB. The weights have not changed. Here GiB means 2³⁰ bytes; do not mix it with decimal GB, or 10⁹ bytes, and check the memory actually available in your GPU environment.

2 × 32 × 8 × 128 × 2 × 8,192 = 1,073,741,824 bytes

When the formula needs adjustment

Sliding-window and chunked attention limit cache growth in the affected layers. Hybrid architectures can store different states in different layers. A shared prefix can sometimes be reused, but do not budget for that saving without confirming the engine configuration.

Weight quantization and KV cache quantization are separate settings. A smaller cache or CPU offloading may save VRAM at the cost of latency. After a change, test quality, response time and freedom from OOM errors at maximum load.

• Read layer count, KV heads and head dimension from the checkpoint.
• Specify cache type and precision, maximum lengths and concurrency.
• Check available memory after loading and peak memory during long prefill.
• Measure the load limit and choose headroom from observations, rather than a universal percentage.