Real-world benchmarks demonstrating performance variances across different GPU cloud infrastructures

Summary

A controlled, apples-to-apples benchmark on CUDO Compute compared three NVIDIA GPUs—H100 SXM, A100, and L40S —under identical software stacks and single-GPU VM configurations.

Key Findings

Using an end-to-end BERT-base masked-LM fine-tune as the workload, the study measured raw throughput (tokens/samples per second) and normalised all results to cost-per-million-tokens.

• H100 combines 4th-gen tensor cores, 3 TB/s NVLink bandwidth, and BF16/FP8 optimisations to deliver the fastest and lowest-cost training and high QPS serving.
• L40S, despite having a lower raw speed, achieves a lower cost-per-token rate compared to the A100 in inference due to a 35% lower hourly rate.

Introduction

Choosing a GPU cloud vendor is no longer a matter of renting the biggest card your budget allows. Transformer workloads have fragmented into a spectrum of use cases—from trillion-parameter pre-training runs to millisecond-latency microservices that serve responses to millions of users per day. Each scenario stresses hardware in different ways, and the same cloud infrastructure can deliver wildly different business outcomes depending on which GPU you deploy.

To clearly illustrate these variations, we conducted a controlled, apples-to-apples benchmark exercise using the open-source Transformers Benchmarks framework. We measured three NVIDIA GPUs — H100 SXM 80GB (Hopper), A100 PCIe 80GB (Ampere), and L40S 48GB (Ada Lovelace) — across both training-centric and inference-centric transformer workloads.

In this article, we will cover our methodology, training and inference result, and provide a roadmap for maximising ROI when selecting GPUs.

Methodology

The configuration below ensures that every result presented in this report can be fully replicated.

1. Test environment

Only one GPU was attached per VM; multi-GPU scaling is outside the scope of this study.

2. Software stack

The container digest was frozen; only the GPU SKU changed between runs.

3. Workload

End-to-end training test – Hugging Face BERT-base masked-LM

• Precision: FP16
• Sequence length: 128
• Batch size: auto-tuned to max VRAM fit (32–64)
• Dataset slice: 4,627 samples (≈86 MB)

This single workload was chosen to reflect a realistic fine-tuning scenario while maintaining a runtime of under two minutes per GPU.

4. Measurement protocol

• Warm-up: 25 iterations to stabilise clocks.
• Timing window: 100 timed iterations; median tokens/s reported.
• Determinism: torch.backends.cudnn.benchmark=False, torch.manual_seed(42).
• No power sampling: Energy tracking reserved for future work.

5. Cost model

CUDO Compute on-demand prices:

The cost-per-sample is calculated by dividing the observed samples by the corresponding hourly rate; no commitment discounts are applied.

6. Limitations

• Single-GPU scope; fabric latency and NVLink scaling not covered
• One workload family (Transformer masked-LM); results should not be generalised to CNNs or diffusion models without further testing.
• No power or thermal data captured.
• Each run was repeated three times; if the relative standard deviation exceeded 5%, a fourth run was performed to replace the outlier.

With the test bed defined, we now turn to the results, comparing raw throughput and cost-normalised performance across H100, A100, and L40S instances.

Training benchmark result

Real-world model training involves two very different stress tests:

• Micro-kernels – e.g. a single BERT encoder layer, which exposes raw tensor-core muscle and memory-bandwidth limits.
• Macro runs – an end-to-end BERT-base masked-LM fine-tune, surfacing dataloader overheads, optimiser cost, and scheduler jitter that ML-Ops teams hit daily.

Below, we walk through both, anchoring the numbers in cost-per-token so that you can see exactly where the budget is allocated.

1. Micro-benchmark headline: BERT layer, seq 512, batch 64

Take-aways:

• Hopper wins on both speed and cost. It's 4th-gen tensor cores, plus 3.36 TB/s memory bandwidth, deliver a 2.15× throughput edge over the A100 and a 23% lower cost-per-token, despite the higher hourly rate.
• L40S stays competitive only on the list price. It’s ~35% cheaper per hour than A100, but the slower tensor cores mean its cost-per-token actually increases by 7% for large-context training.
• A100 is now the “bronze” choice for transformer pre-training—solid, but neither the fastest nor the cheapest on CUDO Compute, especially considering the release of the Blackwell GPUs.

Macro reality-check | Full BERT-base fine-tune (bf16, seq 128)

Why the wider gap?

• CPU ↔ GPU synchronisation: Hopper’s HW-accelerated threadblock-array scheduler keeps streaming multiprocessors (SMs) saturated even when Python dataloaders stall; Ampere idles.
• BF16 optimiser fusion: PyTorch 2.7’s advanced compile path maps directly onto Hopper’s FP8/BF16 tensor-cores, trimming optimiser time by ~30%.
• PCIe tax: The A100 node operates over PCIe Gen-4; the 350 GB/s gap to SXM/ becomes apparent in end-to-end jobs.

Resulting economics

• $0.88 to process 10M tokens on H100
• $2.15 on L40S
• $6.32 on A100

What does this mean for your budget?

Operational notes:

• Memory headroom: Hopper’s 80GB HBM3 enables us to increase BERT batch size from 64 to 96 with no out-of-memory (OOM) errors, reducing epochs per hour by another 1.4 times.
• Gradient checkpointing: Still worth enabling on L40S to maintain utilisation ≥ 95%; on H100, it shaved just 2% off the time-to-accuracy.
• Scheduler simplicity: Since all three GPUs reside within the same CUDO tenancy, you can schedule tasks via a single Terraform module, allowing the price tier to be determined automatically through the use of tags.

Key takeaway:

Every training dollar buys 50–100% more work when you align the GPU SKU to the workload. On CUDO Compute, that’s a one-line terraform change, not a multi-cloud migration. In the next section, we’ll show how the picture flips for latency-critical inference.

Inference benchmark result

For inference workloads, we analysed the benchmark results by doubling the measured train_samples_per_second (since forward-only inference is approximately half the work of a full training step) and converting 128-token sequences to tokens per second. This provides a direct, first-order view of serving economics and highlights the relative performance gaps across SKUs:

Our analysis reveals critical insights for inference performance:

• H100 dominates absolute speed: One H100 SXM card can stream approximately 24,000 tokens per second. This is sufficient to serve a Llama-70B parameter model at around 6 ms/token, enabling real-time chat user experiences even before advanced KV-cache optimisations.
• L40S wins on raw efficiency: Despite being slower than Hopper, the L40S's lower hourly rate makes it the most cost-effective option, delivering the cheapest cost-per-million-tokens. This makes it ideal for bursty allocator pools.
• A100 is a legacy tax: Serving on A100 is almost eight times more expensive per token than on L40S and about seven times slower than Hopper.

Latency nuances

Beyond raw throughput, real-world inference performance is shaped by specific latency characteristics:

• Batch-32 sweet spot: Both Hopper and L40S maintain over 90% utilisation at batch sizes up to 32. Beyond this, queuing delay increases more rapidly on the L40S due to its lack of NVLink peer-to-peer copy.
• Tokeniser masking: Hopper’s FP8 decode path can shave an additional ~12 µs per token when fast RMS-norm kernels are enabled, pushing end-to-end latency below 40 ms for typical 20-token responses.
• Cold-start tales: Container cold-boot on CUDO Compute is primarily network and disk-bound. All three SKUs spin up within approximately 45 seconds from a Terraform apply, allowing scaling policies to remain GPU-agnostic.

Cost-focused deployment patterns

Based on our log-validated benchmarks, here's a playbook for optimising GPU selection on CUDO Compute for various serving patterns:

Quick optimisation tips for ML-Ops teams

To further maximise performance and efficiency:

• Pin torch.compile(fullgraph=True) on Hopper: This optimisation typically gains an additional ~8% in tokens/s by fusing Layer-norm and MatMul operations, requiring no code rewrite.
• Enable NVIDIA TensorRT-LLM on L40S: Activating this can recover approximately 15% throughput, significantly narrowing the speed gap to Ampere while preserving the L40S's price advantage.
• Use weighted round-robin balancer: Implement this in your gRPC balancer to direct long prompts to H100 buckets and short, bursty chat requests to L40S, achieving near-perfect fleet utilisation.

For inference workloads, the freshly crunched logs confirm that H100 maximises raw performance, while L40S minimises cost-per-output. Notably, both new SKUs outperform A100 decisively on every dollar metric within CUDO Compute. Since all three GPUs are accessible under the same CUDO Compute API, swapping SKUs is a single Terraform variable change, not a platform migration, enabling you to fine-tune your latency-versus-cost balance in seconds.

How to plan your GPU selection

The choice of GPU within a cloud environment can significantly impact the cost of compute. Below is a fast-acting playbook you can apply today on CUDO Compute:

For ML-Ops and engineers, here is a checklist you can use:

Sustainability footnote

Since the same job takes half the time on Hopper as on Ampere, the total kWh draw falls proportionally. Pair that with CUDO’s 100% renewable energy commitment, and you get the greenest path to state-of-the-art AI.

Your next three clicks

• Log in to CUDO Portal → pick GPU Catalogue.
• Select the GPU according to the playbook.
• Deploy the GPU and start building.

Ready to cut costs or halve training time? Start a GPU on CUDO Compute now and benchmark your own workloads against the above numbers.

Alternatively, to access large-scale GPU clusters, click here to get started.