Transformer Under the Hood: Visual Attention Guide

Attention Map: Softmax(Q·Kᵀ / √d_k)

Tensor [10, 10]

Color Intensity = Attention Weight Hover any cell to inspect mathematical dot-product

Softmax Temperature (τ): 1.00

Higher τ diffuses attention evenly; lower τ creates sharp, deterministic peaks.

Query Vector Focus Point

Architecture Mode

1. Multi-Head Attention Residual Add + LayerNorm

Projects input tokens into multiple subspaces, allowing simultaneous tracking of syntax, coreference, and positional context.

Why this matters: Enables parallel token interaction without sequential recurrent bottlenecks (O(1) sequential path length).

2. Feed-Forward Network (FFN) d_model → 4×d_model → d_model

Two linear transformations with a non-linear activation (GELU/ReLU) applied identically to each token position.

Why this matters: Stores factual knowledge keys and expands dimensionality to synthesize gathered contextual cues.

3. Residual Skip Connections & LayerNorm x + SubLayer(LayerNorm(x))

Preserves gradient highways across dozens of stacked layers preventing vanishing gradients.