Model Lab

Transformer Self-Attention Explorer

Query: "it" [pos: 7]
Top Attention Target: "animal" (68.4%)
Entropy: 0.94 bits
Token Interaction Graph & Attention Beams Softmax(Q · KT / √dk)
Width = Weight αi,j
Full Attention Matrix A N × N
Head 0
Vector Projections & Context Value zi = ∑ αi,j vj
dmodel = 6
Q:
K:
V:
Z:
Enjoy this tool? Build your own with Super