Temperature: 0.8
Stage 1: Tokenization & Continuous Embeddings
Mapping discrete text words into d_model dimensional vector space.
X_embed = Token_Lookup(Tokens) ∈ ℝ^{N × d_{model}}
Self-Attention Matrix softmax(QK^T / √d_k)
Click any token cell to trace query-to-key weight
Selected: "it" → "animal"
Weight: 0.84
In this sentence, Head 1 resolves the coreference pronoun 'it' to the semantic subject 'animal' rather than 'street'.
Stage Insight
The Transformer breaks strings into tokens, mapping each discrete vocabulary ID to a dense vector (e.g. 512 dimensions). These vectors capture foundational semantic concepts before contextual attention begins.