Long training runs expose infrastructure problems that short jobs never surface. A fine-tuning job that takes four hours on a shared cloud cluster might complete without incident. The same cluster running a pretraining job for six weeks will encounter every structural weakness in the environment, and on shared infrastructure, those weaknesses are not random. They are predictable consequences of how multi-tenant GPU clusters operate.

This is not a case against cloud infrastructure in general. It is a case against using shared, multi-tenant GPU environments for workloads that are fundamentally incompatible with their operational model.

The Failure Modes Are Structural, Not Incidental

The published numbers from production training deployments are instructive. Meta's Llama 3 pretraining on 16,000 GPUs experienced 466 job interruptions over a 54-day period, roughly one interruption every 2.8 hours. At Meta's scale, with dedicated infrastructure and purpose-built fault tolerance tooling, effective training time still came in above 90%. That result required significant engineering investment in automated recovery, checkpoint management, and cluster operations.

For teams running on shared cloud infrastructure without that investment, the numbers look worse. A 32,000-GPU production deployment documented 678 unexpected interruptions, with GPU HBM memory faults, PCIe device failures, and communication timeouts accounting for nearly half of all events. Recovery from a single network failure, detection, node isolation, checkpoint reload, and communication pathway reconstruction, has a documented median time of approximately 68 minutes, representing roughly 12.7% of total job duration when failures occur at typical rates.

The question is not whether failures will happen on a long training run. They will. The question is whether your infrastructure amplifies or contains the damage when they do.

What Shared Cloud Adds on Top of Hardware Failure

Hardware failures are a property of the GPU and the cluster fabric, they occur on any infrastructure at scale. What shared cloud adds on top of that baseline is a set of structural variables that dedicated infrastructure does not have.

Job preemption and scheduling contention. On shared GPU clusters, your training job competes for resources with other tenants. Spot and preemptible instances, the price-competitive options on most cloud providers, can be interrupted at any time when capacity is needed elsewhere. For a training run measured in days or weeks, a preemption event mid-run means reloading from the last checkpoint, reconstructing communication groups, and losing the work done since that checkpoint. On a 16,000-GPU job, that recovery sequence takes over an hour and costs tens of thousands of dollars in wasted GPU hours.

Noisy neighbor effects on interconnect and storage. NVLink bandwidth within a node is not shared across tenants, that isolation holds at the GPU level. But the storage fabric, the network paths between nodes, and the shared switching infrastructure are contested. During checkpoint writes on a shared cluster, aggregate NFS write rates surge as multiple jobs write simultaneously, creating I/O contention that stalls the training pipeline across all nodes. Inconsistent storage throughput during data loading creates GPU pipeline stalls that do not show up in benchmark numbers but compound over a multi-week run.

Driver and software environment inconsistency. Shared cloud infrastructure applies driver updates and system changes across the fleet without necessarily coordinating with your training schedule. A driver update mid-run can change the behavior of communication libraries, affect NCCL performance characteristics, or, in edge cases, break the training configuration you validated before the job started. On dedicated bare metal with a fixed software environment, the configuration you test before day one is the configuration that runs through week six.

Checkpoint egress costs. Long training runs write checkpoints continuously, gradient snapshots, optimizer states, intermediate model artifacts. On metered cloud infrastructure, every checkpoint that leaves the node is a billable egress event. A 70B parameter model checkpoint in FP16 is roughly 140 GB. At typical cloud egress rates, writing checkpoints every 500 steps over a six-week training run accumulates costs that are not always visible in initial compute budget estimates but are real line items by the time the job completes.

What Dedicated Bare Metal Resolves

Dedicated bare metal does not eliminate hardware failures, GPU HBM faults and communication timeouts are hardware-level events that occur regardless of tenancy model. What it eliminates is the additional layer of infrastructure variability that shared cloud introduces.

On dedicated bare metal, your training job is the only job on the cluster. There is no scheduling contention, no preemption risk, and no competing workload consuming storage or network bandwidth. The NVLink fabric, the inter-node network, and the storage I/O are allocated entirely to your run. The configuration you validate before the job starts, driver versions, NCCL settings, checkpoint intervals, data loading pipeline, remains stable through the full duration of the job.

Checkpoint egress is unmetered. A 140 GB checkpoint write costs the same whether you write it once or a thousand times. For a team running a 12-month infrastructure term with continuous pretraining workloads, that difference is a real budget item, not a rounding error.

The operational predictability matters as much as the technical properties. A dedicated bare metal cluster on a 12-month term is infrastructure you can plan against. You know the capacity, the network topology, and the performance characteristics before you commit the compute budget. Training schedules, model roadmaps, and team workflows are built on that foundation. Shared cloud infrastructure, with variable spot availability, unpredictable preemption, and metered costs that shift with utilization, does not provide that foundation for workloads measured in months.

What This Means for Your Training Infrastructure Decision

If your training jobs run in hours, a shared cloud is a reasonable fit. The failure modes described above exist but rarely have time to compound.

If your training jobs run in days or weeks, pretraining a foundation model, continuing pretraining on domain-specific data, large-scale fine-tuning with FSDP or Megatron-LM across multiple nodes, the structural properties of shared cloud work against you. The interruption rate, the storage I/O contention, the egress costs, and the software environment variability are not problems you can engineer around with better checkpoint tooling. They are properties of the infrastructure model itself.

Dedicated bare metal on a 12-month term is not a more expensive version of cloud. It is a different infrastructure model, built for the operational requirements of workloads that shared cloud was not designed to handle. Choosing the right GPU infrastructure provider for long training workloads means choosing one whose operational model matches the workload, not one optimized for short, interruptible jobs at variable spot pricing.

1Legion operates dedicated bare metal GPU clusters on H100 and B300, full-server allocation, no shared tenancy, no egress fees, unmetered bandwidth, on 12-month dedicated terms.

If you are planning a long training run or evaluating infrastructure for a sustained compute workload, talk to an engineer about your requirements today. Talk to an Engineer here.

‍