Where a 70B Model Fits: 80, 96, 141 and 288GB of GPU Memory

Take one model, a 70-billion-parameter LLM in FP8, and load it on four GPUs with 80GB, 96GB, 141GB and 288GB of memory. The weights take 70GB in every case. What differs is what is left: about 10GB, 26GB, 71GB and 218GB. That remainder is where the KV cache lives, so it sets how many sequences the GPU can serve at once and how long they can be. This post works through the 70B model GPU memory requirements at each size, using published architecture figures and no throughput claims.

70GB of weights, four different remainders

Weight memory is parameter count times bytes per parameter: 2 bytes in FP16, 1 in FP8, 0.5 in FP4. For a 70B model that is 140GB, 70GB and 35GB. NVIDIA lists the H100 SXM at 80GB, the RTX PRO 6000 Blackwell at 96GB of GDDR7 with ECC, the H200 at 141GB of HBM3e, and the Blackwell Ultra GPU in the B300 at 288GB of HBM3e.

FP16 weights of 140GB do not fit on an 80GB or a 96GB GPU. On a 141GB GPU they fit with about 1GB to spare, which is not a usable amount. On a 288GB GPU they fit with 148GB left. In practice, then, the FP8 case is the one that all four sizes can run, and it is the one the rest of this post uses. Dropping to FP8 or FP4 trades some accuracy for memory, and how much depends on the model and the quantization method, so test it on your own evaluation set before committing to a size.

0.33MB per token: what the KV cache costs for a 70B model

Weights are a fixed cost. The KV cache is the variable one: every token in every active sequence stores a key and a value vector at each layer, and that memory stays allocated while the sequence is live. Meta's Llama 3 paper gives the 70B architecture: 80 layers, 64 attention heads across an 8,192 model dimension (128 per head), and 8 key/value heads. At 16-bit precision that is 2 x 80 x 8 x 128 x 2 bytes, about 0.33MB per token. An 8,192-token context holds roughly 2.7GB. A 32,768-token context holds roughly 10.7GB.

Divide the remainders from the table above by those two figures:

These are upper bounds. They ignore activation memory, the CUDA context and framework overhead, and they assume a 16-bit KV cache and a model that fits on one GPU. Quantizing the cache stretches them, and allocator inefficiency shrinks them. On an 80GB GPU the overhead matters most: the 10GB that remains is less than one full 32K-token sequence, so an FP8 70B model on 80GB is a short-context deployment unless the cache is quantized or the model is split across GPUs.

From 3 to 81 sequences at 8K: where each size stops

The table shows a pattern more than a ranking. Each step up in memory roughly triples the room for sequences, but the steps are not interchangeable, because the right size is set by the context length and concurrency a deployment needs, not by the model alone.

• 80GB: holds a 70B model in FP8 and serves short contexts at low concurrency. Past that, a second GPU is needed, which brings the interconnect into the sizing.
• 96GB: the same model with enough room for a handful of 8K sequences, or a couple at 32K. Our post on the RTX Pro 6000 Blackwell node covers this case, including QLoRA fine-tuning.
• 141GB: room for about 26 sequences at 8K or about six at 32K on one GPU. NVIDIA lists the H200 at 4.8TB/s of memory bandwidth, against 3.35TB/s for the H100, so the extra capacity comes with more bandwidth. Our B300 vs. H200 post covers when this size is enough.
• 288GB: about 20 concurrent sequences at 32K on one GPU, and enough capacity that a 405B model fits at FP4. NVIDIA lists 8TB/s per GPU. Our B300 post works through the 405B case.

Capacity is only half of a deployment. Bandwidth sets how fast tokens are generated once the model is loaded, and the memory sizes above come paired with different bandwidths. The tables in this post tell you what fits, and your own workload tells you how fast it runs.

The sizing rule: weights + KV cache + headroom must fit in one GPU

To check a model against any of these sizes, pick the precision, multiply parameters by bytes per parameter, then estimate the KV cache as 2 x layers x KV heads x head dimension x bytes per value, multiplied by your target context length and concurrency. Leave room for activations and framework overhead. If the total fits, the model is a per-GPU workload and the node gives you one replica per GPU. If it does not, the next question is how the model gets split across GPUs, and that depends on the interconnect as much as on memory.

Talk to an engineer about which GPU memory tier fits your models.