Layer 2 — The silicon that powers AI
agent37.club
Everything above depends on what gets fabricated in silicon.
AI training = massive matrix multiplications. This is embarrassingly parallel. GPUs were built for exactly this.
Three resources: compute (CUDA/Tensor cores), memory capacity (how much HBM), memory bandwidth (how fast you can read it).
AI workloads are almost always memory-bound, not compute-bound.
A tensor core performs a 4×4 matrix multiply-accumulate in a single clock cycle.
D = A × B + C
// Where A, B, C, D are 4×4 matrices
// One clock cycle. One operation.
// A CUDA core would need 64 multiply + 64 add.
H100 Tensor Core performance:
Lower precision = 2x throughput per step down. FP8 is sufficient for most inference. Training usually uses FP16/BF16.
This is why quantization works — you trade tiny accuracy loss for massive speedup.
| GPU | Year | HBM | Capacity | Bandwidth | FP8 TFLOPS |
|---|---|---|---|---|---|
| A100 | 2020 | HBM2e | 80 GB | 2.0 TB/s | 624 |
| H100 | 2022 | HBM3 | 80 GB | 3.35 TB/s | 3,958 |
| H200 | 2024 | HBM3e | 141 GB | 4.8 TB/s | 3,958 |
| B200 | 2025 | HBM3e | 192 GB | 8.0 TB/s | 9,000 |
| Vera Rubin | 2026 | HBM4 | 288 GB | 22 TB/s | 50 PFLOPS* |
* Vera Rubin: 50 PFLOPS FP4 inference. 336B transistors. NVL72 rack: 3.6 EFLOPS.
Notice: memory capacity grew 1.8x (A100→H200) while compute grew 6.3x. The gap is the memory wall.
Regular DDR5: chips spread horizontally on a board. One channel ≈ 50 GB/s.
HBM: stack 8–12 DRAM dies vertically, connected by thousands of through-silicon vias (TSVs).
During inference, every token’s attention keys & values must be stored in GPU memory. This is the KV cache.
More HBM = longer context windows = more concurrent users = more revenue per GPU.
Same compute. The only difference is memory: 80 GB → 141 GB HBM3e.
That extra 61 GB means ~1.8x more concurrent users or ~1.8x longer context. Same chip, dramatically more revenue.
Memory capacity — not compute — determines how many users a GPU can serve.
The gap between compute growth and memory bandwidth growth
over the last decade
Growing 2–3x per generation (~every 2 years). Tensor core throughput doubles reliably.
Growing ~1.3x per generation. Physics makes it hard — signal integrity, power, thermals.
This is THE fundamental bottleneck. We have more compute than we can feed with data. Every AI optimization (quantization, FlashAttention, KV cache compression) is ultimately fighting the memory wall.
Compute (FP8 TFLOPS) — indexed to A100 = 1x
Memory Bandwidth — indexed to A100 = 1x
Compute grew 14.4x. Bandwidth grew 4x. The ratio keeps getting worse.
of all advanced chips (<7nm) are made by one company: Taiwan Semiconductor Manufacturing Company.
One earthquake. One geopolitical crisis. One company.
No TSMC = no advanced GPUs = no AI scaling. This is the single biggest concentration risk in the global economy.
Next gen: High-NA EUV ($380M+) for 2nm and below.
ASML makes the machines → TSMC uses them to make chips → NVIDIA designs the chips → Microsoft/Google buy them.
Every advanced AI chip on Earth passes through an ASML machine. There is no alternative.
| Node | Year | Cost / Wafer | Transistor Density | Who Can Afford It |
|---|---|---|---|---|
| 28nm | 2011 | ~$3,000 | ~50M/mm² | Many fabs |
| 7nm | 2018 | ~$10,000 | ~100M/mm² | TSMC, Samsung, Intel |
| 5nm | 2020 | ~$16,000 | ~170M/mm² | TSMC, Samsung |
| 3nm | 2022 | ~$20,000 | ~290M/mm² | TSMC, Samsung |
| 2nm | 2025 | ~$30,000 | ~500M/mm² | TSMC (mainly) |
The economics are shifting. Cost per transistor stopped falling at 3nm. Each node now costs more per transistor than the last. Moore’s Law isn’t dead, but the economics of Moore’s Law are.
CUDA ecosystem built over 15+ years:
You can build a competitive GPU. You cannot replicate 15 years of software ecosystem, library support, developer muscle memory, and Stack Overflow answers.
This is why NVIDIA can charge $30–40K per GPU and still have a 2-year backlog.
| MI300X | H100 | |
|---|---|---|
| HBM | 192 GB | 80 GB |
| Bandwidth | 5.3 TB/s | 3.35 TB/s |
| FP16 | 1,307 TFLOPS | 1,979 TFLOPS |
| Price | ~$15K | ~$30K |
The value bet: 2.4x the memory at half the price. If ROCm catches up, AMD is a serious contender for inference workloads.
| Company | Chip | Focus | Key Advantage |
|---|---|---|---|
| TPU v5p | Training + Inference | Tightly integrated with JAX/TensorFlow; powers Gemini | |
| Amazon | Trainium2 | Training | 40% better price/perf vs GPU (claimed); AWS integration |
| Microsoft | Maia 100 | Inference | Co-designed with Azure; liquid-cooled from day one |
| Meta | MTIA v2 | Inference | Optimized for ranking & recommendation models |
The trade-off: custom chips are cheaper per inference — but only for the specific workloads they’re designed for. NVIDIA GPUs are general-purpose. Custom silicon is purpose-built.
Claims: 4x fewer GPUs to train same MoE model vs Blackwell. 10x reduction in inference token cost.
Companion CPU: Vera — 227B transistors, 88 custom Arm cores, 1.5 TB LPDDR5x. The CPU-GPU boundary is blurring.
NVIDIA’s cadence: new architecture every year. The moat isn’t one chip — it’s the relentless pace.
| AI Capability | Constrained By | Chip Bottleneck |
|---|---|---|
| Model size | HBM capacity | How many GB on the GPU |
| Context window | KV cache memory | HBM capacity + bandwidth |
| Inference cost | GPU price / throughput | $/TFLOP and memory wall |
| Training time | Compute throughput | Tensor core TFLOPS |
| Batch size | Memory capacity | HBM per GPU |
| Multi-GPU scaling | Interconnect bandwidth | NVLink / InfiniBand |
The chip layer sets the ceiling for everything above. No software trick can overcome a hardware limit — you can only work around it more cleverly.
| Company | Market Cap | Why It Matters | Key Risk |
|---|---|---|---|
| NVIDIA | ~$2T+ | Dominant GPU + CUDA ecosystem | Priced for perfection |
| TSMC | ~$800B+ | Makes 90% of advanced chips | Taiwan geopolitics |
| ASML | ~$350B+ | Monopoly on EUV lithography | Cyclical capex |
| SK Hynix | ~$100B+ | HBM market leader (~50%) | Memory cyclicality |
| AMD | ~$200B+ | Value alternative, 192GB MI300X | ROCm software gap |
The thesis: AI demand is structural, not cyclical. But the stocks are priced for massive growth. The edge is understanding which bottleneck is binding and when it shifts.
Next week: The Supply Chain — DRAM, NAND, fabs, and the memory super cycle
agent37.club