Why memory bandwidth — not compute — is the real bottleneck for AI
agent37.club
During single-request LLM inference, GPUs are typically less than 1% utilized on compute. The tensor cores sit idle, starving for data. The bottleneck isn’t math — it’s moving bytes.
Arithmetic intensity = FLOPs ÷ Bytes transferred. When this is below the GPU’s ops:byte ratio, you’re memory-bound. For LLM decode at batch size 1, it’s always memory-bound.
// Generating one token from a 70B model // 1. Read all weights from HBM bytes_read = 70e9 * 2 // 140 GB (FP16) // 2. Compute: one matmul per layer flops = 2 * 70e9 // ~140 GFLOP // 3. Arithmetic intensity intensity = flops / bytes_read // = 140 GFLOP / 140 GB // = 1 FLOP/byte // H100 needs 295 FLOP/byte to // saturate compute. // We're at 1. 295x under-utilized.
| GPU | Peak TFLOP/s | BW (TB/s) | Ridge Point |
|---|---|---|---|
| A100 | 312 | 2.0 | 156 ops/byte |
| H100 | 989 | 3.35 | 295 ops/byte |
| B200 | ~2,250 | 8.0 | ~281 ops/byte |
At batch size 1, LLM decode has arithmetic intensity of ~1. The ridge point for an H100 is 295. You’re operating at 0.3% of the way to compute saturation.
For LLM inference, buying a faster GPU doesn’t help much — you need more memory bandwidth. This is why HBM matters more than FLOPS for serving AI.
Batch size matters: Increasing batch size improves arithmetic intensity for FFN layers — at batch 500+, FFN becomes compute-bound on an H100. But attention remains memory-bound at all practical batch sizes because the KV cache grows with the batch.
Stacked DRAM dies, through-silicon vias, and the interposer
Bandwidth = bus width × clock speed. HBM’s 1024-bit bus is 16× wider than DDR5’s 64-bit channel. Even at lower clock speeds, HBM delivers far more bandwidth.
// Bandwidth comparison // DDR5 (8 channels, server) bw = 64 bits × 8 ch × 4800 MHz = 307 GB/s // HBM3e (single stack) bw = 1024 bits × 9.2 Gbps = 1,180 GB/s // ~1.2 TB/s // H200: 6 stacks of HBM3e total = 6 × 1.2 TB/s = 4.8 TB/s // B200: 8 stacks of HBM3e total = 8 × 1.0 TB/s = 8.0 TB/s // 26x DDR5!
Each HBM stack is under 1mm thin, contains thousands of microscopic TSV drill-holes, and must handle up to 15W of heat with no hotspots exceeding 85°C.
| Stack Height | Per-Layer Yield | Stack Yield |
|---|---|---|
| 8-hi (HBM3e) | 95% | 66% |
| 12-hi (HBM3e) | 95% | 54% |
| 16-hi (HBM4) | 95% | 44% |
Going from 8-hi to 12-hi drops yield by 15–20%. At 16 layers, nearly half of all stacks are defective.
Current HBM3e yields are estimated at 50–60% in production. Samsung famously failed multiple rounds of NVIDIA qualification testing for HBM3e, partly due to yield issues.
SK Hynix’s proprietary Mass Reflow Molded Underfill process has 3× more thermal bumps than Samsung/Micron’s TC-NCF approach, providing much better heat dissipation. This is a key reason for their market lead.
From HBM2e to HBM4 — doubling bandwidth every generation
| Generation | BW / Stack | Capacity / Stack | Bus Width | Key GPUs |
|---|---|---|---|---|
| HBM2e | 460 GB/s | 16 GB (8-hi) | 1024-bit | A100, MI250 |
| HBM3 | 820 GB/s | 24 GB (8-hi) | 1024-bit | H100 |
| HBM3e | 1.2 TB/s | 36 GB (12-hi) | 1024-bit | H200, B200, MI300X |
| HBM4 | 2.0 TB/s | 48 GB (8-hi) | 2048-bit | Vera Rubin, MI400 |
The trend: Each generation roughly doubles bandwidth. HBM4 also doubles the bus width from 1024 to 2048 bits — this is not a drop-in replacement. It requires entirely new interposer designs and memory controllers.
288 GB HBM4 at 22 TB/s memory bandwidth. 8-hi stacks. NVL144 launching 2026; Rubin Ultra NVL576 in 2027.
432 GB HBM4 at 19.6 TB/s. 12-hi stacks on CDNA 5. MI455X and MI430X shipping 2026.
Hybrid bonding becomes “indispensable” for 20+ layer stacking — micro-bumps can’t scale that far without unacceptable yield loss and thermal issues.
Three companies. One bottleneck. A $30B+ market.
HBM market share as of Q2 2025. Samsung’s share collapsed after failing NVIDIA’s HBM3e qualification repeatedly.
FY2025: 97.15 trillion won revenue (~50% YoY growth). Operating profit: 47.21 trillion won. Overtook Samsung in annual profit for the first time ever.
Failed NVIDIA’s HBM3e qualification multiple times. Recently passed for 12-layer. Now racing to deliver HBM4 samples early to recover.
| Metric | FY2025 |
|---|---|
| Revenue | 97.15 trillion won |
| Operating profit | 47.21 trillion won |
| Net profit | 42.95 trillion won |
| Q4 revenue (record) | 32.83 trillion won |
| HBM % of DRAM revenue | >40% |
| HBM revenue YoY growth | 2× |
SK Hynix’s MR-MUF packaging process provides a thermal advantage that Samsung and Micron haven’t matched. This is why they consistently win NVIDIA’s qualification tests.
HBM prices jumped >60% between 2024 and 2025. Samsung and SK Hynix are planning ~20% price hikes for 2026 as demand for H200s and ASICs remains insatiable.
The shift of DRAM wafer capacity to HBM is causing shortages in DDR5 for servers. Server memory prices could double by 2026. Samsung raised 32GB DDR5 prices from $149 to $239 in just two months.
Why you can’t buy enough GPUs — and HBM is a big reason why
DRAM supplier inventories fell from 13–17 weeks (late 2024) to 2–4 weeks (October 2025). The entire chain is stretched thin.
Memory shortage expected to persist through at least late 2027 (per SK Hynix guidance).
From A100 to Vera Rubin: 11× bandwidth improvement in four GPU generations. Compute grew ~8× in the same period. The industry is clearly prioritizing bandwidth over raw FLOPS — because the memory wall is the binding constraint.
HBM is reshaping the entire memory industry
NVIDIA’s stringent qualification tests give it enormous leverage. Failing qualification (as Samsung did with HBM3e) means losing billions in revenue.
Note: Samsung is racing to deliver HBM4 samples early to regain share lost during the HBM3e qualification failures.
The question for investors: Is HBM demand genuinely structural (like NAND in smartphones) or a cyclical AI capex boom that will revert? The supply sold-out-through-2027 signal suggests structural.
The memory wall is the most important constraint in AI infrastructure.
agent37.club