Resources 
You can buy two identical B200 clusters and get entirely different throughput out of them in production. Buyers typically plug the static GPU count and nominal interconnect speeds from a datasheet into their performance models to forecast throughput. The problem is when actual workloads run on the physical hardware, operational realities introduce constraints that theoretical models simply miss, creating a persistent gap between the published specifications and the sustained performance the cluster ultimately delivers.
This is the case for every GPU cluster, across every generation, and it traces almost entirely to how each environment is architected, tuned, and operated, rather than to the chips inside it. A spec sheet describes a GPU in isolation, under ideal conditions, doing nothing but compute.
Production throughput is a property of the whole system under sustained load: the fabric carrying traffic between nodes, the collective-communication library moving gradients, the storage feeding data fast enough to keep the GPUs busy, the power and thermal envelope holding under weeks of continuous draw, and the operational tooling that keeps a single failure from stalling the rest.
The hardware sets the floor, and what the buyer actually gets is decided in the work done between procurement and production.
Why identical GPUs deliver different performance
The intuition that identical hardware should yield identical results is reasonable but mistaken, because the spec sheet measures only the chip, which is the one thing that does not vary, and says nothing about the things that do.
The clearest published illustration comes from DeepSeek: training DeepSeek-V3 ran on 2,048 NVIDIA H800 GPUs, hardware built on the same Hopper architecture as the H100 but with NVLink bandwidth reduced from 900 GB/s to 400 GB/s for export compliance.
That single difference in interconnect reshaped the entire training strategy. The team avoided tensor parallelism because it was inefficient under the constrained NVLink, built a bespoke bidirectional pipeline scheme to overlap computation and communication, and relied on eight 400G InfiniBand NICs per node to compensate for the loss of intra-node bandwidth.
The chip was nearly the same, and the configuration determined what the cluster could actually do. Even after that hardware-aware co-design, third-party analyses estimate the run landed at around 23% model FLOPs utilization, a figure the DeepSeek team never had to disclose and that most buyers never think to ask about.
That 23% is not fixed by the hardware. Other large runs landed elsewhere: ByteDance’s MegaScale system, training a 175B-parameter model on 12,288 GPUs, reported 55.2% MFU, more than double DeepSeek’s figure on comparable-era hardware.
The spread between those two numbers is engineering: how the fabric is laid out, how collectives are scheduled, how stragglers are caught, and how the data pipeline is fed. Two organizations running similar hardware ended up with a 2x difference in how much useful work the GPUs actually did.
What causes GPU cluster divergence in training
Three mechanisms separate one training fleet from another in practice.
The first is collective communication, since distributed training spends a large fraction of each step moving gradients between GPUs, and the library that handles it, NCCL, is highly sensitive to configuration. In one published analysis, a misconfigured NCCL setup on 8xH100 training a 70B model consumed 20 to 30% of total step time on communication; tuning the right environment variables for the topology cut that to 5 to 8%, a swing that can be up to 10 to 25% of training cost at scale.
If NCCL auto-selects the wrong network interface for a GPU group, every inter-node packet takes an extra hop through the PCIe switch, adding microseconds per collective call that compound into seconds per step.
NVIDIA’s own multi-node tuning guidance for GB200 systems makes the same point in the other direction: the optimal number of communication blocks depends on the specific topology and must be benchmarked rather than assumed. The hardware ships identical; the communication efficiency is configured afterward.
The second is reliability, where the spec sheet misleads most: it describes a chip that never fails, while real clusters fail constantly. Meta’s Llama 3 training run used 16,384 H100 GPUs over 54 days and logged 419 unexpected interruptions, roughly one every three hours, more than half of which were GPU or HBM3-related.
Because synchronous training is only as fast as its slowest participant, a single faulty GPU can require restarting the entire job, and a single straggler, a GPU running slightly slow rather than failing outright, silently drags thousands of others down with it.
Two identical clusters running the same job will diverge sharply depending on whether the operator can detect and evict that straggler in minutes or discovers it after a wasted day. Meta held above 90% effective training time across that run, but that number was an operational achievement built on straggler detection and fast checkpointing, not a property the hardware delivered on its own.
The third is the physical envelope, where the same Llama 3 run recorded 1 to 2% throughput variation from midday temperature swings alone, as dynamic voltage and frequency scaling responded to ambient heat. Power delivery, cooling headroom, and rack-level thermal design are not on the GPU spec sheet, and they decide whether a chip sustains its rated clocks under weeks of continuous load or quietly throttles.
How execution affects inference throughput
Inference does not escape any of this, because MFU in training and tokens per second in serving are two names for the same thing, which is the share of the GPU’s theoretical throughput you actually capture.
Both leak for the same reason: a GPU stalled waiting on the rest of the system, data that has not arrived, a collective that has not finished, or saturated memory bandwidth, rather than doing calculations. Training and inference just expose different stalls, and both come down to the same whole-system-under-load question the spec sheet does not account for.
The serving stack is a configuration choice the hardware knows nothing about. On the same GB200, the same model can run materially faster or slower depending on the inference engine. Published comparisons put the gap between vLLM and TensorRT-LLM at roughly 20 to 40% in throughput on identical hardware, with one benchmark showing Llama 2 70B in FP8 on a single H100 moving from 85 to 95 tokens per second under vLLM to 110 to 130 tokens per second under a compiled TensorRT-LLM engine.
The size of that gap varies with the model, batch size, sequence length, and precision, and these benchmarks are best read as directional rather than precise. What remains consistent across all of them is that the runtime decision alone increases throughput by double digits on unchanged hardware.
Precision changes the same equation, and the clearest first-party demonstration comes from NVIDIA’s own benchmarking. On the same setup, Llama-3.1 8B in FP16 handled about 256 concurrent users at roughly 72 tokens per second per user before breaching a 50-token-per-second service level; switching to the FP8 checkpoint let the same hardware serve 512 concurrent users at 66 tokens per second per user. A checkpoint swap doubled the machine’s effective capacity.
The same holds at the smallest scale, down to individual flags. Switching a multimodal model’s vision encoder from tensor-parallel to data-parallel mode, a single configuration change, yields a 10 to 45% throughput improvement with negligible memory cost, by replacing dozens of per-layer synchronization operations with a single gather.
That particular figure comes from one vendor’s tuning guide, but nothing in the mechanism is vendor-specific, as the gain comes from how the parallelism was arranged and not from whose GPU it ran on. None of that is predictable, and all of it determines what the cluster actually produces.
Does newer hardware fix the performance gap?
It would be reasonable to assume that the gap is a problem of older or constrained hardware and that the latest generation resolves it, but it does not, because the divergence persists across generations and the newest hardware shows it as clearly as the oldest.
The clearest case is within the Blackwell generation, because it changes the interconnect without changing the chip. A GB200 NVL72 is built from the same B200 GPUs as a standalone B200 server; the difference is how they are interconnected, 72 GPUs in a single NVLink domain versus an eight-GPU island.
That change in interconnect produces very different results depending on the workload and how it is run. On benchmarks published by SemiAnalysis, the two are within a few percent of each other on a model like gpt-oss 120B, where the B200 is actually marginally faster on some configurations.
On a large mixture-of-experts model like DeepSeek R1, the gap opens dramatically, and it is not a single figure: at one measured operating point the GB200 NVL72 delivered about 4.4x the per-GPU throughput of the B200, while at a different point on the same curve the two were within roughly ten percent of each other.
The reason is those larger models fire heavy all-to-all traffic between experts, and the 72-GPU NVLink domain absorbs it where the eight-GPU island cannot, forcing the overflow onto slower Ethernet. Same chip, and the realized throughput per GPU ranges from near-parity to several times higher, depending on whether the workload and its operating point match how the GPUs are connected.
The pattern extends even to the CPU. The host processor feeding the GPUs is one factor we expect to matter more over the next cycle: single-threaded performance there sets how fast work reaches the accelerators, and NVIDIA’s newer systems pair notably stronger CPUs with them for exactly that reason.
Read more on this here: Land • Power • Compute
When GPU tuning and execution won’t help
The argument has a boundary, and naming it matters because execution quality does not override the hardware’s limits. A workload that is genuinely compute-bound will not run faster because the cluster is well-operated, and no amount of tuning can lift a job above the silicon it runs on.
The GB200’s interconnect advantage is a case in point: on a workload that fits comfortably within a single node and never needs to spread traffic across the wide NVLink domain, that advantage collapses to near-parity, which is exactly what the curve shows at the operating points where the two systems converge.
Where the hardware is the binding constraint, it decides the outcome. The claim is narrower and more useful than “execution beats hardware”: it is that between two clusters built on the same hardware, the realized throughput is decided by architecture, configuration, and operations, and that the distance between the floor the hardware sets and the result the buyer gets is large, often a factor of two or more, and almost always underestimated at procurement.
So the question to put to a vendor is not “how many GPUs” or “which generation,” because those are the floor, and every vendor quotes the same floor. The questions that separate them are the ones about the other three sides of the balance: what storage bandwidth actually reaches the GPUs under load, how the fabric is monitored and what happens when a straggler appears, what sustained throughput looks like after thermal steady state rather than in the first hour, and who owns the tuning after the hardware is racked. Answering those questions in specifics is the most important evaluation you can get, and whoever cannot answer them is selling a floor and letting you discover the rest in production.
Hardware sets the floor, execution sets the result
The hardware is a floor that any competent buyer can read off a datasheet. The result is an operational outcome that depends on the work done between signing the purchase order and running the workload in production.
In training, that means the fabric tuned to the topology, the collectives configured for the job, the storage feeding data fast enough to keep the GPUs busy, the stragglers caught before they cost a day, and the thermal envelope held under load.
In serving, it means the inference stack and precision are matched to the workload, and the software stack is mature enough to realize what the silicon promised. Two identical clusters diverge because that work is never identical. The hardware is bought; the throughput is built.
If you are standing up a cluster, inheriting one that is underperforming, or trying to work out why your utilization does not match your invoice, talk to our team about what it would take to close the gap.
