In MLPerf Inference v4.0, NVIDIA published results for the H200 on Llama 2 70B. The post that reports them gives three different answers to how much faster the H200 is than the H100: up to 28%, 43% and 45%. A fourth figure, 37%, appears further down the same post and does not describe the H200 at all.

None of these numbers is wrong. Each one answers a different question, and the conditions that separate them are written in the results tables and the submitters’ notes. This guide sets out how to read MLPerf results, and any other GPU benchmark, by asking five questions before a number goes into a sizing decision. The H200 result is the worked example throughout.

1. Which division was the result submitted in?

MLPerf Inference has two divisions. In the closed division, every submitter runs the same reference model, so results compare hardware and software on equal terms. In the open division, submitters may change the model, for example by pruning or retraining it, and must report what they changed.

The 37% comes from the open division. It describes an H100 submission that applied structured sparsity to Llama 2 70B and made the model 37% smaller. Offline throughput rose 33% over the closed submission, and the model still met the stricter of the benchmark’s two accuracy targets. That is a result for a modified model on an H100. It says nothing about the H200 against the H100.

What to check: the division label and, for open-division rows, the notes on what was changed. If you plan to serve the model as published, compare closed-division results.

2. Which scenario does the number describe?

The data center results are reported in two scenarios. In the offline scenario, every query is available at the start and the benchmark measures throughput. In the server scenario, queries arrive at random intervals and the system must keep tail latency under a limit set for each benchmark.

NVIDIA’s 43% is the server result and 45% is the offline result for the same H200 configuration. Here the gap is two points. For other models and systems it can be much wider, because batching freely is easier than meeting a latency bound.

What to check: match the scenario to how your workload arrives. Batch generation, evaluation runs and embedding jobs look like offline. A user-facing endpoint with a latency target looks like server.

3. At what power and cooling was it measured?

The 28% compares the H200 and the H100 at the same thermal design power of 700 W. The 43% and 45% come from an H200 system with a custom thermal design that let each GPU run at 1,000 W. The extra power added 11% in the server scenario and 14% offline on top of the 700 W H200 result.

MLPerf reports measured power only for submissions that enter the power category, and it measures the whole system at the wall. MLCommons does not measure or validate TDP settings.

What to check: the power limit and cooling of the submitted system, and whether the server you will run on can sustain the same. On dedicated servers, the power cap is set by the server configuration and by the power each rack is contracted to draw.

4. Which software, and from which round?

The same NVIDIA post reports that the H100 ran GPT-J 2.4 times faster offline and 2.9 times faster in the server scenario than in the previous round, using TensorRT-LLM. The GPU did not change. The inference software did.

A comparison between results from different rounds, or from different inference stacks, mixes hardware gains with software gains. MLPerf also labels each system as Available (for purchase or rent), Preview (to be available by the next round) or research, development and internal.

What to check: the round, the software stack listed in the system description and the availability category. To isolate the hardware, compare results from the same round and stack.

5. Does the submitted system match the node you will run on?

A result describes the submitter’s system: GPU count, memory per GPU, interconnect, CPUs, power cap, cooling and software. The node you rent may differ on any of these.

Memory per GPU decides what fits before throughput matters. Our team worked through that arithmetic for a 70 GB model on 80, 96, 141 and 270 GB GPUs in 70 GB of weights, four GPUs, and for a full node in 8 x 270 GB: what fits on a B300 node.

What to check: the system description file for GPU count, memory, NVLink domain, network and power cap, compared line by line with the node in your proposal.

A checklist for the next result

  • Division: closed or open, and what was changed in open
  • Scenario: offline, server or another
  • Power and cooling: TDP setting, custom cooling, measured system power if reported
  • Software and round: inference stack, MLPerf round, availability category
  • System: GPU count, memory per GPU, interconnect and power cap

Where a benchmark stops and an acceptance report starts

A benchmark tells you what a submitter’s system did under defined conditions. It does not tell you how the servers you rent will behave.

That is what burn-in is for. Before handover, every 1Legion cluster passes five tests: HW-1 measures NCCL collective throughput with nccl-tests, HW-2 tests every InfiniBand or RoCE rail with ib_write_bw, HW-3 covers GPUDirect RDMA and NVMe storage, HW-4 checks NVLink and NVSwitch bandwidth with NVbandwidth, and HW-5 is a 24-hour soak under a representative inference load at a sustained power cap. The customer receives the acceptance report. We do not offer trials. The burn-in is the proof.

Read how burn-in and acceptance work, or request a quote with the hardware, number of servers, term and start date.