A GPU cluster can pass a quick check and still fail its first long training run. A degraded network rail, a GPU that throttles after an hour at full power or a storage path that quietly falls back to the CPU will not show up in a smoke test. They show up later, in a job that slows down or stops.
Every 1Legion cluster passes NVIDIA’s hardware validation tests, HW-1 to HW-5, before handover. This post explains what each one measures and what it is designed to catch, using the documentation of the tools themselves. It picks up where Five questions to ask before you use a GPU benchmark to pick a node ended: where a benchmark stops and an acceptance report starts.
HW-1: NCCL collective throughput, measured as bus bandwidth
Training jobs spend much of their communication time in collectives such as all-reduce. HW-1 runs nccl-tests across the cluster and checks that GPUs in different servers exchange data at the throughput the test requires.
Bus bandwidth is the figure to compare. The nccl-tests performance notes define algorithm bandwidth as data size divided by time, a figure that tends to fall as the number of GPUs grows. Bus bandwidth corrects for that. For all-reduce the correction factor is 2(n−1)/n, which on one eight-GPU node is 1.75. The corrected result should “reflect the speed of the hardware bottleneck,” whether that is NVLink, PCIe or the network, so it can be compared with the hardware’s rated bandwidth.
Where the network supports it, HW-1 also runs with SHARP, which performs part of the reduction inside the switches.
What it catches: shortfalls that only appear at cluster scale, such as a topology misconfiguration or a switch that limits throughput between racks.
HW-2: every InfiniBand or RoCE rail, one at a time
An NVIDIA HGX B300 cluster built to our delivery specification gives each GPU an 800 Gb/s backend port, configured as one 800, two 400 or four 200 Gb/s links, on a rail-optimized, non-blocking fabric dedicated to one customer. That is 6.4 Tb/s of east-west bandwidth per eight-GPU node. HW-2 measures RDMA write bandwidth on every rail with ib_write_bw, and each rail has to perform on its own.
Testing rail by rail is a matter of arithmetic. A collective across hundreds of links can lose one link’s worth of bandwidth and still look close to normal in aggregate. Tested alone, a rail with a faulty transceiver, a damaged cable or a wrong port setting stands out at once. In a running job, the same rail slows every collective that crosses it.
HW-3: GPUDirect RDMA and NVMe storage
GPUDirect RDMA lets the network card read and write GPU memory directly, without a copy through host memory. When it is not working, traffic still flows, but through the CPU, and throughput drops without an error message.
HW-3 checks that path for each GPU and network card pair. NVIDIA’s GPUDirect RDMA testing guide runs ib_write_bw with a specific GPU as the memory target, so the measurement is of the direct path between that GPU and its network card. HW-3 also runs storage smoke tests, including NVMe read and write on the local data drives, at least 9.2 TB of raw NVMe per node in the specification, which jobs use for datasets and checkpoints.
HW-4: NVLink and NVSwitch inside each node
Inside an HGX B300 node, the eight GPUs share one NVLink domain through NVSwitch. HW-4 measures it with nvbandwidth, NVIDIA’s tool for measuring “bandwidth and latency for various memory copy patterns across different links.”
nvbandwidth runs GPU-to-GPU copies pairwise, one-to-all and all-to-one, using both the copy engines and the streaming multiprocessors. According to NVIDIA’s DCGM documentation, its copy tests also check that the copied data matches the source. HW-4 checks two things: that every expected NVLink link is up, and that bandwidth matches what the topology should deliver. A missing or degraded link shows up as one GPU pair running below the rest.
HW-5: 24 hours under inference load at a sustained power cap
The first four tests measure peak throughput. HW-5 measures whether the cluster holds it. It runs a representative inference load on every GPU for 24 hours at a sustained power cap.
Some faults appear only after hours at full power: a GPU that throttles as it heats, a cooling problem in one rack position, memory errors that build up under load, or a power supply that trips under sustained draw. A short test passes all of them. A 24-hour soak gives them time to appear before your first job does.
What the acceptance report contains
The acceptance report records the HW-1 to HW-5 results for your cluster and becomes its baseline. The test tools are public, so your own engineers can rerun them at any point and compare against the report. The hardware inventory, from racks and nodes down to GPUs, network cards, drives, switches, ports, transceivers and cables, is available as a machine-readable export.
When a node fails a test
A node that fails is quarantined, repaired or replaced, and tested again. The cluster is handed over only after every node passes all five tests. The same rule continues after handover: a GPU with a fatal fault is quarantined at once, and a node suspected of silent data corruption is held in quarantine, under the operating commitments.
Why there is no trial
We do not offer trials. The burn-in is the proof. The same goes for demos, proofs of concept and short-term or month-to-month rentals.
A trial usually means one node for a few days. One node cannot show a rail fault between racks, a cluster-wide collective shortfall or a rack position that runs hot. The acceptance report covers the cluster you will run on, every node and every rail, after 24 hours under load.
Read how burn-in and acceptance work, the delivery specification the results are measured against, or request a quote with the hardware, number of servers, term and start date.