In a transformer layer, each input token produces three vectors: Query (Q), Key (K), and Value (V). To compute attention weights, the model evaluates the dot-product similarity between the query of token i and the keys of all tokens j, scales by 1/√dₖ to stabilize gradients, and applies Softmax to generate a normalized probability distribution.
Different attention heads specialize in distinct linguistic patterns: some resolve ambiguous pronouns to their referents, others connect adjectives to adjacent nouns, and others capture broad relational context.