Training and inference are different problems. They share hardware, they share frameworks, and they are often discussed together, but the infrastructure constraints that matter for each are not the same. A GPU that is excellent for training a 70B model may be a poor fit for serving it at production traffic levels.
The NVIDIA B300 (Blackwell Ultra) was designed with inference economics in mind. Understanding what that means in practice, beyond the headline throughput numbers, is what determines whether it is the right infrastructure for your serving workload.
Why inference has different hardware requirements
Training is a sustained, memory-intensive workload where the primary bottleneck is compute throughput and inter-GPU communication. You run a defined job, it completes, and the result is a model artifact. Latency variability within the job matters less than total time to convergence.
Production inference is different in every dimension that matters operationally. Requests arrive continuously and concurrently. Latency must stay within bounds that users actually experience, a response that takes 30 seconds is a failed user interaction, not a long training step. The KV cache grows with context length, which means memory pressure scales with traffic patterns rather than model size alone. And cost-per-token is a live metric, not a post-hoc accounting exercise.
These constraints put specific pressure on three hardware characteristics: memory capacity, compute efficiency at low batch sizes, and sustained throughput under concurrent load. The B300 addresses all three, but the reasons why are worth understanding in detail.
What 270 GB of HBM3e changes at serving time
For most serving workloads, the binding constraint is not compute, it is memory. The model weights occupy a fixed portion of VRAM; everything else goes to the KV cache that stores attention state for in-flight requests.
On an H100 with 80 GB of HBM3, a 70B parameter model in FP16 consumes approximately 140 GB, meaning it cannot fit on a single GPU at all. Teams either run tensor parallelism across two GPUs, paying inter-GPU communication overhead on every forward pass, or quantize the model to FP8 to reduce the memory footprint. Both options involve trade-offs.
On a B300 with 270 GB, the same 70B model fits on a single GPU with roughly 130 GB remaining for KV cache. At a 32K context window, a single request can consume 8 to 16 GB of KV cache depending on the model architecture. That remaining headroom supports about a dozen concurrent long-context requests without cache eviction, without the tail latency spike that occurs when hot KV blocks get swapped to CPU memory and must be reloaded.
For reasoning models, architectures that generate extended chain-of-thought sequences before producing an output, this is the difference between viable and not. A 128K context window on a reasoning model under concurrent load will exhaust the H100's available KV cache headroom quickly. On the B300, that headroom exists without architectural compromises. We work through the numbers model by model in 8 x 270GB: what fits on a B300 node.
FP4 compute and what it means for cost-per-token
NVIDIA lists the 8-GPU HGX B300 at 108 petaFLOPS of dense FP4 compute (144 petaFLOPS sparse), 13.5 petaFLOPS per GPU. The practical consequence of that figure shows up in cost-per-token at scale, which is ultimately the metric that determines fleet economics for any team running a production inference API.
Hopper-generation GPUs such as the H100 and H200 have no FP4 Tensor Core support, so a model served in FP4 on the B300 runs at a precision those nodes cannot use. For teams where fleet cost is a meaningful operating expense, serving the same traffic load on fewer GPUs is where that difference shows up, and it compounds at scale.
FP4 support in production requires TensorRT-LLM 0.17 or higher, or vLLM with FP4 quantization enabled. The model weights and activations need to be quantized accordingly. For most transformer architectures, the accuracy impact is negligible for inference tasks. The serving stack update is real work, but it is an inference stack configuration change, not a retraining of the model.
Latency consistency under concurrent load
Throughput benchmarks measure peak performance under controlled conditions. Production inference operates under variable concurrent load, with requests of different lengths arriving unpredictably. The latency profile under those conditions is what actually determines user experience.
On shared cloud GPU infrastructure, latency consistency is degraded by factors the hardware cannot compensate for: storage I/O variability during model weight loading or KV cache offload, and network overhead on the data path between the inference node and the load balancer. These sources of variance are structural, they exist regardless of how capable the GPU is.
On dedicated bare metal, none of those variables apply. The GPU's full memory bandwidth and compute throughput are available to your inference workload at all times. There are no neighboring tenants competing for network bandwidth. Storage throughput is consistent. The tail latency you observe in load testing is the tail latency you get in production, which is the only condition under which latency SLAs are meaningful.
For the B300 specifically, this matters more than it did for prior generations. At 7.7 TB/s of memory bandwidth and 13.5 petaFLOPS of dense FP4 compute per GPU, the B300's peak capability is high enough that any external source of contention becomes a proportionally larger drag on realized throughput. Dedicated infrastructure is not just operationally cleaner, it is how you actually capture the hardware's inference performance in production.
B300 bare metal inference at 1Legion
1Legion's HGX B300 servers are open for allocation, served from Toronto, as whole 8-GPU servers on 36, 48 or 60 month terms: dedicated infrastructure, no shared tenancy, no egress fees, and no hyperscaler overhead between your inference stack and the hardware.
If you are evaluating B300 for a production inference deployment or want to discuss your specific serving requirements, talk to an engineer.