LLM Tokens Explained: Interactive Tokenizer

1Live tokenizer

Type anything:

0tokens

0characters

0chars / token

Price tier:

$0cost of this text

0%of a 128k context

Approximate GPT-style tokenizer (BPE heuristic). Real tokenizers like tiktoken differ slightly, but the counts land within a few percent on typical English.

Drag to orbit · blocks = your tokens

Each 3D block is one token from your text — width tracks its character length, and the two colors alternate token boundaries. More tokens, taller the spiral. That spiral is literally what you pay for.

2What exactly is a token?

A token is a subword unit — bigger than a character, usually smaller than a word. Modern LLMs (GPT, Claude, Llama) build their vocabularies with byte-pair encoding (BPE): start from raw bytes, then repeatedly merge the most frequent adjacent pair until you reach a target vocabulary size (GPT-4's cl100k_base has ~100,256 tokens; GPT-4o's o200k_base ~200k).

Step 0 lower — every byte is its own token

Merge 1 lower — "lo" and "er" are frequent pairs

Merge 2 lower — "low" earned its own token

Result lower → 2 tokens; a common word like the → 1 token (leading space included!)

Because merges are frequency-driven, common English text compresses well: about 4 characters ≈ 1 token, or 1 token ≈ 0.75 words (100 tokens ≈ 75 words). "Antidisestablishmentarianism" appears rarely, so it shatters into ~6 chunks; "the" appears billions of times, so it's a single token.

Worked example: a 1,500-word essay ≈ 1,500 ÷ 0.75 = ~2,000 tokens. At $2.50 per million input tokens that's $0.005 to read — half a cent.

3Why some text costs more

Same idea, very different token bills
Input	Chars	≈ Tokens	Why
"the cat sat"	11	3	All common words → 1 token each
"1234567890"	10	4	Digits merge in short 1–3 digit runs, so arithmetic is genuinely harder for LLMs
"antidisestablishmentarianism"	28	6	Rare word → shattered into subwords
"こんにちは" (hello, Japanese)	5	3–5	Non-Latin scripts got fewer merges; some CJK chars cost 1+ tokens each
Burmese / Khmer sentence	~40	40–100+	Low-resource scripts can cost up to ~10× English per meaning — a real equity issue in API pricing

Numbers: "1234567" is not one token — GPT-4-class tokenizers chunk digits in groups of up to three, e.g. 123·456·7. This is one reason LLMs slip on long arithmetic.
Spaces matter: " hello" (with leading space) and "hello" are different tokens. Capitalization too: "Hello" ≠ "hello".
Code: indentation-heavy code tokenizes surprisingly well because tokenizers learned merged runs of spaces (e.g. 4 spaces = 1 token).

4Context windows & pricing

Context window = token budget

A model's context window is the maximum tokens it can attend to at once — prompt plus its reply. Typical sizes: 8k → 128k → 200k → 1M tokens.

128k tokens ≈ 96,000 words ≈ a 300-page novel
200k tokens ≈ ~150,000 words
Attention cost grows with context, which is why long context is priced at a premium

Pricing is per token, twice

APIs bill input tokens (your prompt) and output tokens (the reply) at different rates — output usually costs 3–5× more because generation is one forward pass per token.

Worked example: 10k-token prompt + 1k-token answer at $2.50 in / $10 out per 1M tokens:

10,000 × $0.0000025 = $0.025
1,000 × $0.00001 = $0.010
Total ≈ $0.035

Mental model: a token is the atom of LLM economics. Vocabulary design decides how many atoms your text needs; the context window caps how many fit in the room; the price list charges you per atom in and per atom out.