Long training runs expose infrastructure problems that short jobs never surface. A fine-tuning job that takes four hours on a shared cloud cluster might complete without incident. The same cluster running a pretraining job for six weeks will encounter every structural weakness in the environment, and on shared infrastructure, those weaknesses are not random. They are predictable consequences of how multi-tenant GPU clusters operate.
This is not a case against cloud infrastructure in general. It is a case against using shared, multi-tenant GPU environments for workloads that are fundamentally incompatible with their operational model.
The failure modes are structural, not incidental
The published numbers from production training deployments are instructive. Meta's Llama 3 pretraining on 16,000 GPUs experienced 466 job interruptions over a 54-day period, roughly one interruption every 2.8 hours. At Meta's scale, with dedicated infrastructure and purpose-built fault tolerance tooling, effective training time still came in above 90%. That result required significant engineering investment in automated recovery, checkpoint management, and cluster operations.
For teams running on shared cloud infrastructure without that investment, each failure costs more. GPU memory faults, PCIe device failures and communication timeouts all stop a synchronous job, and recovering from one means detecting it, isolating the node, reloading the last checkpoint and rebuilding communication between the remaining nodes. The longer the run, the more of those recoveries it pays for.
The question is not whether failures will happen on a long training run. They will. The question is whether your infrastructure amplifies or contains the damage when they do.
What shared cloud adds on top of hardware failure
Hardware failures are a property of the GPU and the cluster fabric, they occur on any infrastructure at scale. What shared cloud adds on top of that baseline is a set of structural variables that dedicated infrastructure does not have.
Job preemption and scheduling contention. On shared GPU clusters, your training job competes for resources with other tenants. Spot and preemptible instances, the price-competitive options on most cloud providers, can be interrupted at any time when capacity is needed elsewhere. For a training run measured in days or weeks, a preemption event mid-run means reloading from the last checkpoint, reconstructing communication groups, and losing the work done since that checkpoint. On a 16,000-GPU job, that recovery sequence takes over an hour and costs tens of thousands of dollars in wasted GPU hours.
Noisy neighbor effects on interconnect and storage. NVLink bandwidth within a node is not shared across tenants, that isolation holds at the GPU level. But the storage fabric, the network paths between nodes, and the shared switching infrastructure are contested. During checkpoint writes on a shared cluster, aggregate NFS write rates surge as multiple jobs write simultaneously, creating I/O contention that stalls the training pipeline across all nodes. Inconsistent storage throughput during data loading creates GPU pipeline stalls that do not show up in benchmark numbers but compound over a multi-week run.
Driver and software environment inconsistency. Shared cloud infrastructure applies driver updates and system changes across the fleet without necessarily coordinating with your training schedule. A driver update mid-run can change the behavior of communication libraries, affect NCCL performance characteristics, or, in edge cases, break the training configuration you validated before the job started. On dedicated bare metal with a fixed software environment, the configuration you test before day one is the configuration that runs through week six.
Checkpoint egress costs. Long training runs write checkpoints continuously, gradient snapshots, optimizer states, intermediate model artifacts. On metered cloud infrastructure, every checkpoint that leaves the node is a billable egress event. A 70B parameter model checkpoint in FP16 is roughly 140 GB. At typical cloud egress rates, writing checkpoints every 500 steps over a six-week training run accumulates costs that are not always visible in initial compute budget estimates but are real line items by the time the job completes.
What dedicated bare metal resolves
Dedicated bare metal does not eliminate hardware failures, GPU HBM faults and communication timeouts are hardware-level events that occur regardless of tenancy model. What it eliminates is the additional layer of infrastructure variability that shared cloud introduces. What it adds is an operator accountable for the hardware: on 1Legion clusters, fatal GPU faults are quarantined immediately and NCCL hangs are correlated with fabric events, as set out in our operating commitments.
On dedicated bare metal, your training job is the only job on the cluster. There is no scheduling contention, no preemption risk, and no competing workload consuming storage or network bandwidth. The NVLink fabric, the inter-node network, and the storage I/O are allocated entirely to your run. The configuration you validate before the job starts, driver versions, NCCL settings, checkpoint intervals, data loading pipeline, remains stable through the full duration of the job.
Checkpoint egress carries no fees. A 140 GB checkpoint write costs the same whether you write it once or a thousand times. For a team running continuous pretraining on a multi-year term, that makes checkpoint frequency an engineering decision rather than a billing one.
The operational predictability matters as much as the technical properties. A dedicated bare metal cluster on a fixed term is infrastructure you can plan against. You know the capacity, the network topology, and the performance characteristics before you commit the compute budget. Training schedules, model roadmaps, and team workflows are built on that foundation. Shared cloud infrastructure, with variable spot availability, unpredictable preemption, and metered costs that shift with utilization, does not provide that foundation for workloads measured in months.
What this means for your training infrastructure decision
If your training jobs run in hours, a shared cloud is a reasonable fit. The failure modes described above exist but rarely have time to compound.
If your training jobs run in days or weeks, pretraining a foundation model, continuing pretraining on domain-specific data, large-scale fine-tuning with FSDP or Megatron-LM across multiple nodes, the structural properties of shared cloud work against you. The interruption rate, the storage I/O contention, the egress costs, and the software environment variability are not problems you can engineer around with better checkpoint tooling. They are properties of the infrastructure model itself.
Dedicated bare metal on a fixed term is not cloud with a longer contract. It is a different infrastructure model, built for the operational requirements of workloads that shared cloud was not designed to handle. Choosing the right GPU infrastructure provider for long training workloads means choosing one whose operational model matches the workload, not one optimized for short, interruptible jobs at variable spot pricing.
1Legion rents dedicated bare metal GPU clusters built on the B300: full-server allocation, no shared tenancy, no egress fees, on 36, 48 or 60 month dedicated terms.
If you are planning a long training run or evaluating infrastructure for a sustained compute workload, talk to an engineer about your requirements.