RTX 4090 or L40S: which GPU should you rent for AI?

How the RTX 4090 and L40S differ

The RTX 4090 is a GeForce GPU; the L40S is a data-centre GPU. Both use NVIDIA Ada Lovelace architecture. When renting, look beyond the product category to memory, support for your software and the complete server configuration.

The UZ Dynamics catalogue includes the L40S but not the RTX 4090. NVIDIA’s RTX 4090 specifications describe its Founders Edition or reference design; cards from other manufacturers may differ. Confirm the server configuration, access and support terms for the specific offer.

24 or 48 GB: will the workload fit?

Model file size is not peak GPU memory usage. Weights share memory with working data and buffers; training also needs gradients and optimiser state. Nominal capacity does not mean all 24 or 48 GB is available to your code: the environment may occupy or reserve some memory.

For an initial estimate, calculate weights alone. Exactly 8 billion parameters stored at 2 bytes each require 16 billion bytes: 16 GB, or about 14.9 GiB. The example excludes the KV cache, activations, metadata and other overhead, so it does not establish that the workload fits on an RTX 4090.

Compare peak usage at the settings you need. When it approaches available capacity, a short successful run is insufficient: test long inputs and expected demand. Options for reducing memory pressure include changing settings, suitable quantization, moving some data to system RAM or choosing another configuration. Retest output quality and runtime for each option.

Example weights only: 8,000,000,000 × 2 bytes = 16 GB ≈ 14.9 GiB

• Explore VRAM and the KV cache

For LLMs, account for context and concurrent requests

Text generation needs more than loaded weights. The KV cache stores intermediate attention data, and its size depends on model architecture, sequence lengths, requests being processed and the implementation. A model can handle a short conversation yet run out of memory with a longer context or heavier demand.

Include the RTX 4090 in your test if the model, precision and workload fit in available memory. Test the L40S if memory for the model, context or concurrent requests becomes the constraint. At startup, vLLM logs the KV cache size in tokens and an estimate of how many requests of the configured maximum length (max_model_len) it can serve concurrently. Compare those lines on both GPUs with the same settings.

Keep the model and version, weight format, input and output lengths, engine and request concurrency fixed. Measure time to first token, generation speed and overall throughput. A tokens/s number without test conditions does not describe the experience of a conversation or service behaviour under load.

• Choose a configuration for LLM inference

Image generation and fine-tuning need different tests

For Stable Diffusion, FLUX or another generator, specify the exact model, pipeline version, resolution, images per run, precision and extra components. Compare matching generation steps and output requirements. Generating one image does not establish that the same workflow will run at a higher resolution or with additional models.

For example, FLUX.1 [schnell] is a 12-billion-parameter transformer: its BF16 weights alone take about 24 GB, almost all of an RTX 4090’s nominal memory. The text encoders and VAE add about 10 GB more. So on one RTX 4090 this pipeline will not fit entirely in BF16 without moving components to the CPU or quantization. On an L40S, the roughly 34 GB of weights leave headroom; only a test shows whether it is enough for your resolution and number of images.

Diffusers offers ways to reduce memory usage, including moving components to the CPU and processing images in tiles. According to its documentation, offloading parts of a model to the CPU (sequential CPU offloading) dramatically reduces memory usage but is extremely slow and often impractical; offloading whole models is faster but saves less. So an RTX 4090 with these settings and an L40S without them are two different execution configurations: the speed difference cannot be attributed solely to the GPU.

Assess fine-tuning separately from inference. Full fine-tuning updates model weights; LoRA trains additional adapters, while QLoRA combines adapters with a quantized base model. These approaches change memory needs without removing activation and training costs. Check example length, batch size, optimiser and memory-saving settings. A model that generates responses on an RTX 4090 may not support fine-tuning there with the settings you need.

• Check LoRA and QLoRA requirements

Why more memory does not mean more speed

Runtime depends not only on memory capacity but also on computation, data movement and software implementation. Advertised TFLOPS or CUDA core counts do not translate directly into tokens/s, images per minute or training time.

The two GPUs’ specifications pull in different directions. The RTX 4090 has higher memory bandwidth: 1,008 GB/s against the L40S’s 864 GB/s. The L40S has higher peak dense BF16 Tensor Core throughput: 362.05 TFLOPS against 165.2. NVIDIA describes processing the input (prefill) as a highly parallel operation that effectively saturates the GPU, and token generation as memory-bound: its latency is dominated by how fast weights, the KV cache and activations move from memory, not by how fast the computation runs. So long inputs, large batches and training are more likely to gain from the L40S’s compute headroom, and long responses to single requests from the RTX 4090’s memory bandwidth. These are hypotheses to test: the outcome depends on the model, precision, engine and load.

If the workload fits comfortably on both GPUs, measure the difference. NVIDIA calls batching the simplest way to improve GPU utilization and throughput: requests share one model, so the memory cost of its weights is spread across them. The L40S’s extra memory lets you test a larger batch; that can change both throughput and response latency. Batch processing and an interactive service can call for different settings.

Check the entire rented resource. The CPU, system RAM, disk and network contribute to preparing and feeding data. If time goes into loading a model or reading a dataset, a different GPU won’t shorten it. Save software versions and test settings alongside your measurements.

Two RTX 4090s do not become one 48 GB GPU

Two RTX 4090s have 48 GB of nominal memory in total, but each retains its own 24 GB. Software must support distributing the model or work across devices. With two independent model replicas running entirely on GPU, without CPU offload, each needs to fit on its own device.

Neither the RTX 4090 nor the L40S supports NVLink. That does not prevent multi-GPU work, but how the model is split affects the result. vLLM’s documentation advises pipeline parallelism instead of tensor parallelism when the GPUs in a node lack NVLink, as the L40S does, for higher throughput and lower communication overhead. If you split a large model across several GPUs with Accelerate’s device_map, only one of them works at a time. Compare a specific multi-GPU configuration against one L40S, including cost and software support, rather than adding up memory alone.

Compare cost per result, not just per hour

Use the same job: for example, generating one image set at specified settings or processing one request set. After checking output quality, compare duration and charges. A lower hourly rate can come with a longer run; a more expensive hour does not prove that the job costs more.

Budget for the complete configuration, billable preparation and waiting, storage and data transfer. For a continuously served model, include expected demand and response-time requirements. Check current rates and GPU availability in the actual offer.

• Calculate costs using your rates

How to test the choice on your workload

Prepare a small sample resembling your future workload and set acceptable quality and runtime in advance. Record enough detail to reproduce the test.

• Fix the model, software versions, precision, settings and server configuration. Check applicable terms for using the model and software.
• Separate environment startup and model loading from warm execution. Repeat the test several times and record peak memory, duration and errors.
• Test expected demand: long inputs, concurrency, required resolution or batch size. For a service, measure latency under that load.
• Compare the results against total cost. Before a longer rental, confirm configuration availability, access, billing, storage and support terms.

• Estimate your workload’s GPU requirements