LoRA and QLoRA: plan your GPU configuration

Separate the base model from its adapters

LoRA freezes base weights and trains added low-rank matrices. QLoRA combines adapters with a quantized base model. The original paper stores base weights in 4 bits; that does not make all computation and trainable parameters 4-bit.

Full fine-tuning updates base weights and has a different gradient and optimiser-state budget. An inference estimate cannot serve as a training estimate without accounting for these extra components.

Calculate one adapter as an example

For a linear layer with input dimension din and output dimension dout, LoRA adds approximately r × (din + dout) parameters, excluding bias and any additional trainable modules. With din = dout = 4,096 and r = 16, that is 131,072 parameters: 262,144 bytes, or 0.25 MiB, if only these weights are stored in BF16.

This is one layer, not the full training budget. Include every target layer, gradients, optimiser state, possible master copies, activations and base weights. Raising the rank from 16 to 64 quadruples the adapter parameter count in this example.

LoRA parameters per layer ≈ r × (din + dout)

Record the settings that affect the estimate

Fix the checkpoint, target_modules, rank, base-weight and compute precision, optimiser, maximum sequence length and micro-batch per GPU. In PEFT, choosing all-linear for QLoRA targets more modules than adding adapters only to query and value layers.

Check gradient checkpointing, gradient accumulation, example packing and distribution across GPUs. They change the balance between memory and time. Accumulation builds an effective batch over several steps; it does not hold that entire batch in VRAM at once.

Run a short but complete trial

Use representative examples, including the longest ones. Test a complete training step with backward and optimiser update, saving and reloading the adapter, and evaluation with the settings intended for the main run.

The QLoRA result for a 65B model on one 48 GB GPU belongs to the authors’ experimental setup. It does not guarantee that every 65B checkpoint, sequence length and batch will fit your card. This guide contains no UZ Dynamics benchmark results.

• Record peak VRAM, step time and processed token count.
• Save CUDA, PyTorch, Transformers, PEFT and quantization backend versions.
• Compare quality on held-out examples before and after training.
• Provide the settings and test results with the required rental duration to clarify the proposal.