Context Window Tokenizer BPE v4.0

Subword tokenization, entity density & prompt context footprint workbench

Input Text & Model Parameters
0 Total Tokens
0 Total Characters
0.00 Chars / Token
0 Entity Tokens
Context Window Utilization (128K) 0.02%
Subword Token Stream & Categorization
Tokens: 31
Chars: 161
Chars/Token: 5.19
Entity Tokens: 12
Context Window: 0.02%
Enjoy this tool? Build your own with Super