Neural Architecture Tradeoff Workbench — Transformer vs SSM vs CNN vs GNN

Task Benchmarks Preset

Model Parameters Live Controls

Primary Architecture

Sequence Length ($N$) 4096 tokens

Hidden State / Model Dim ($d$) 128

Attention Heads / SSM Recurrent Expansion 8

Simulation Step Scrubber Step 0

Transformer context quadratic memory wall vs SSM hidden state loss risk

Scaling Metrics Transformer

Memory Footprint

128 MB

O(N²) quadratic

Compute FLOPs

2.1 GFLOPs

Attention Matrix

Context Capacity

Bounded

KV Cache Limit

Recurrent State

None

Full Sequence

Memory Scaling Curves (MB)

Karpathy Verdict Recommendation

Recommended: Hybrid SSM-Attention

Combine SSM's unbounded context linear $O(N)$ scanning with selective sparse Transformer attention for exact multi-hop recall.