Foundation Model Training Bottlenecks: Memory Bandwidth, Not GPU Count

Foundation model training infrastructure depends on GPU memory bandwidth and interconnect, not GPU count. Here's the data behind it