Deep Learning Architecture Comparison

Why Transformers Replaced RNNs

Explore the computational and architectural bottlenecks that caused Recurrent Neural Networks (RNNs/LSTMs) to be superseded by Self-Attention Transformers: sequential dependency vs. constant $O(1)$ path length and hardware parallelization.

Simulation Parameters

12 tokens
41224
Token 1

Token where initial context originates.

75% / step

Simulates vanishing gradient / memory decay across recurrent hops.

Information Flow & Signal Retention

RNN Path (Sequential) Transformer (Direct Attention)
RNN Training Steps
12 serialized
$O(N)$ sequential operations. Cannot parallelize forward or backward pass across time.
Transformer Training Steps
1 parallel op
$O(1)$ sequential operations. All tokens processed concurrently on GPU/TPU matrix cores.
Signal at Final Token (RNN)
4.2%
Exponential decay: $\text{Signal} = \gamma^{N - k}$. Causes vanishing gradient on long sequences.
Signal at Final Token (Self-Attn)
100.0%
Direct query-key connection $A_{i,j} = \text{softmax}(QK^T/\sqrt{d})$. Maximum path length = 1.

Key Architectural Paradigms: RNN vs. Transformer

Why Recurrent Models Hit a Wall

Sequential Bottleneck: RNNs update hidden state via $h_t = f(h_{t-1}, x_t)$. Computing $h_t$ strictly requires $h_{t-1}$, preventing parallel matrix operations across time steps during training.

Information Loss: Long-range relationships must pass through every intermediate hidden vector as an information bottleneck, leading to vanishing or exploding gradients despite LSTM/GRU gating.

Why Self-Attention Won

Massive Parallelism: Because transformers discard recurrence in favor of positional encodings and self-attention, all $N$ tokens are processed simultaneously in unified matrix multiplications across GPU tensor cores.

Constant Path Length: Every token can attend directly to any prior or arbitrary token with a path length of $O(1)$, unlocking effective context windows scaling from thousands to millions of tokens.