🧠 Transformer Architecture & Live Tensors
Params: 1,648
1. TOKENIZER & EMBEDDING (d_model=16)
[2, 16]
2. MULTI-HEAD SELF-ATTENTION (Head 0 Scores)
Softmax(Q · Kᵀ / √d_k)
Attention Weights Matrix
Head 0 (d_k=8)
Hover/Click matrix cell: Q[1] · K[0]ᵀ / 2.83 → Softmax = 0.30
3. FEEDFORWARD (MLP 16→64→16) + RESIDUAL
LayerNorm & ReLU
Residual stream accumulates attention output + FFN projection before next block.
4. OUTPUT UNEMBEDDING & PREDICTED TOKEN
Top Logit: Next Token
"sat" (p=0.482)
⚡ In-Browser Training & Compute Scaling
Loss: 2.079Initial Loss
2.079
Current Loss
2.079
Trained Steps
0
Learning Rate
0.050
Steps 0 → 10
HARDWARE ESTIMATOR (Paul Graham 17yo Premise)
Compute Budget
EXPORT VALIDATED PYTHON / PYTORCH SCRIPT
# micro_gpt.py - Autonomous Mini-Transformer
import torch, torch.nn as nn
from torch.nn import functional as F
class TinyGPT(nn.Module):
def __init__(self, vocab_size=8, d_model=16, n_heads=2, n_layers=2):
super().__init__()
self.token_emb = nn.Embedding(vocab_size, d_model)
self.pos_emb = nn.Embedding(6, d_model)
# Multi-Head Attention & MLP Layers