Read the explanation
The source generates synthetic sixteen-dimensional query, key and value vectors from token hashes. It then adds hand-written lexical biases. Use three tokens: it, animal, street. Inspect it at index zero in the first head. Scores are query dot key divided by square root of sixteen and temperature. This is arithmetic from the actual source, not a trained transformer or a request to any backend model. Increase temperature from zero point five to two while keeping all vectors fixed and using the bidirectional mask. Three markers follow the stable softmax probabilities for it, animal and street. The source subtracts the maximum score before exponentiating and normalizes by their sum. Higher temperature reduces score differences after division; the distribution moves toward a more even weighting, though it is not exactly uniform at finite temperature. Each temperature gives three nonnegative weights whose sum is one. Differences here come from seeded vectors and hand-written noun and pronoun biases, not learned semantic understanding. The video uses the exact source token sequence and head; it does not generalize these weights into claims about an actual language model. Keep the same token sequence, first head and query at index zero. The bidirectional mode permits all three keys. The causal mode excludes keys with indices greater than the query index. Here animal and street are future positions, so both are masked. The only remaining key is the token it itself. Again increase temperature from zero point five to two. The gold curve is bidirectional self-attention weight. The cyan reference is the causal self-attention weight, exactly one throughout. Under the causal mask, the two future weights remain zero. The source writes a sentinel score and directly sets masked exponentials to zero, rather than allowing a tiny leakage probability. Temperature cannot spread probability to a key that this mask excludes. With only one allowed key, softmax assigns all weight to it. The curves compare two fixed mask modes, rather than interpolating between masks or inventing partial masking states. This is a discrete eligibility rule followed by numeric normalization. The source inspector combines attention weights with value vectors. For each output dimension it sums weight times that key’s value component. Consider the first output component in the same bidirectional example. The three value components are fixed. The gray reference lines show these three source values on a signed axis. Increase temperature from zero point five to two once more. Recompute the three source softmax weights, then their weighted sum of first value components. The marker follows this exact inspector output expression. Because the weights are nonnegative and sum to one, the output stays between the smallest and largest value components. It is not a predicted next token or a trained model answer. Only weighting changed; query, key and value vectors remained fixed. The source has four synthetic heads, but these scenes isolate its first head and one output component. It has no learned projection matrices, feed-forward stack or actual language inference in this demonstration. The video preserves the original controls and complete query, key, value exports while explaining the actual synthetic attention arithmetic.