Throughput / GPU
191.4 tok/s
Bandwidth Limited
HBM Utilization
37.9%
Capacity: 640 GB Total
NAND Stall Penalty
0.0 ms
No Storage Spill
Total System Memory BOM
$15,200
HBM $12.8k | NAND $2.4k
2D Silicon Interposer & Multi-Tier Memory Topology
Stacked HBM3e (DRAM)
PCIe NVMe (NAND Flash)
HBM Bandwidth-Bound System: Stacked DRAM interposer bandwidth dominates token speed; NAND bulk storage offload introduces 100x IO stall penalty if weights spill over HBM boundary.
All weights and KV cache fit cleanly within primary HBM3e stacks. Interposer TSVs operate at optimal bus throughput.
Economic Breakdown: Non-Commodity Stacked DRAM vs Commodity Bulk NAND
"Storage is Commodity, HBM is Not. Be careful choosing between NAND capacity vs DRAM... particularly when stacked as HBM."
Structural Economic Dynamics:
• HBM (Stacked DRAM): Requires advanced 2.5D/3D silicon interposers & Through-Silicon Vias (TSVs). Yield bottlenecks limit supply, maintaining high gross margins (~65%).
• NAND Flash: Standardized multi-layer planar/3D die stacking without complex silicon interposers. Highly elastic supply leads to rapid commodity pricing cycles.
• HBM (Stacked DRAM): Requires advanced 2.5D/3D silicon interposers & Through-Silicon Vias (TSVs). Yield bottlenecks limit supply, maintaining high gross margins (~65%).
• NAND Flash: Standardized multi-layer planar/3D die stacking without complex silicon interposers. Highly elastic supply leads to rapid commodity pricing cycles.
Workload & Model Specs
Model Parameters
70 B
Quantization
16-bit (FP16)
Context Window Length
8,192 tok
Batch Size
32
Hardware Stack Configuration
GPU Count
8 GPUs
HBM Capacity / GPU
80 GB
HBM Bandwidth / GPU
3.35 TB/s
NAND Flash Read Bandwidth
14.5 GB/s
Actions & Export