LLM Tokenizer & Polyglot Byte-Subword Diagnostic

Analyze multi-script BPE subword fusion, UTF-8 byte leakage, and cross-boundary tokenization across modern LLM vocabularies.

31 chars | 35 UTF-8 bytes
Presets:

Diagnostic Telemetry Boundary Active

Total Tokens
8
Byte Efficiency
4.38 B/T
Scripts Detected
ASCII + Han (CJK)
Cross-Script Leak
Detected (ASCII/CJK)
Subword Insight: Standard BPE tokenizers group ASCII spaces with non-Latin Unicode blocks (e.g. "both需要"), creating cross-script subword fusion.

Token Chip Deconstruction View

Click any token chip below to inspect its UTF-8 byte structure, Unicode codepoints, and BPE merge rank.

ASCII/English
CJK (Han)
Cross-Boundary

UTF-8 Byte Array & Subword Matrix

Token #0 Selected
SELECTED TOKEN ID 2432
BYTE COUNT 3 Bytes
UNICODE SCRIPT ASCII + Han

Subword Breakdown Specs

Byte-level Inspection
Idx Subword Text Hex Stream Codepoints
Cross-Script Fusion Analysis Moderate Leak

The subword boundaries merge non-ASCII unicode characters directly into standard Latin token sequences due to missing word-boundary split rules in BPE pre-tokenization.