How subword tokenization differs across English, code, and CJK text
Modern BPE tokenizers (such as OpenAI's o200k_base family, Anthropic's Claude 5 tokenizer, and SentencePiece-based Gemini vocabularies) split text into byte or character subwords rather than whitespace words. Plain English prose averages roughly 3.8 to 4.2 characters per token, whereas JSON keys, indentation, stack traces, and URLs introduce punctuation boundaries that push density closer to 2.5 to 3.2 characters per token.
Chinese, Japanese, and Korean scripts encode fewer characters per token—typically 0.6 to 1.1 tokens per character depending on vocabulary coverage—and Anthropic's newer Claude 5 tokenizer can produce up to 25%–30% more tokens than older Sonnet 4.6 vocabularies on dense code or multilingual inputs.
| Input content type | Typical token density | Budgeting recommendation |
|---|---|---|
| Plain English prose | ~0.75 tokens / word (~4 chars / token) | Safe for initial sizing; add 10% headroom |
| TypeScript / Python / JSON schemas | 1 token per 2.5–3.3 characters | Minify repeated tool schemas and strip comments |
| Chinese / Japanese / Korean (CJK) | ~0.7–1.1 tokens per character | Verify exact usage.prompt_tokens on one real call |
| Multi-turn agent transcripts | Compounds on every tool turn | Place static instructions first to maximize cache reads |