Tokenization in LLMs — How AI Models Read Text

What Is a Token?

A token is the basic unit of text that LLMs process. Tokens are not the same as characters, words, or syllables — they sit somewhere in between. A common English word like "tokenize" might be split as token + ize (2 tokens), while "cat" is a single token.

Examples:

"Hello, world!" → 4 tokens: Hello, ,, world, !
"supercalifragilistic" → 6 tokens
"OpenClaw" → 2 tokens: Open, Claw
Emojis, Unicode characters, and non-English text can use 2–6 tokens per character

Byte Pair Encoding (BPE)

Most LLMs use BPE tokenization, introduced by Sennrich et al. (2016). The algorithm works by:

Starting with a vocabulary of individual bytes (256 entries)
Counting the most common adjacent byte pairs in a large training corpus
Merging the most frequent pair into a new vocabulary entry
Repeating until the vocabulary reaches the target size (typically 50k–100k tokens)

The result: common English words and subwords have compact representations, while rare strings are split into more tokens.

Token Counts by Model

Model	Vocabulary size	Context window	Tokenizer
GPT-4o, GPT-4o-mini	~100k	128k tokens	tiktoken cl100k
Claude 3.5 (Anthropic)	~100k	200k tokens	Anthropic BPE
Gemini 1.5 Pro	~256k	1M tokens	SentencePiece
Llama 3 (Meta)	128k	128k tokens	tiktoken BPE

Cost and Context Implications

API pricing is always per token, charged separately for input and output:

A 1,000-word document ≈ 1,300–1,500 tokens
A 500-step agent run with verbose tool results can easily use 200k+ tokens
Output tokens typically cost 3–5× more than input tokens

Context window usage affects latency too — a fuller context window means more work for the model's attention mechanism, increasing response time.

Token Counting with the OpenClaw SDK

from openclaw import count_tokens

text = "Summarise the key points of the Python 3.13 release notes."
n = count_tokens(text, model="gpt-4o")
print(f"{n} tokens")  # 13 tokens

# Count an entire message list
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is BPE tokenization?"},
]
n = count_tokens(messages, model="gpt-4o")
print(f"{n} tokens")  # ~24 tokens (includes message overhead)

Tips to Reduce Token Usage

Shorten system prompts — remove redundant instructions; a 50-token system prompt is usually better than a 500-token one
Prefer concise tool descriptions — each tool schema is included in every request
Use structured data over prose — a JSON object with 5 fields uses fewer tokens than describing the same data in sentences
Truncate large tool results — if a web page returns 10,000 words, extract only the relevant section before adding to context
Use smaller models for simple tasks — GPT-4o-mini costs 30× less than GPT-4o for the same token count
Enable context summarisation — compress old context instead of passing it verbatim

Key takeaways:

Tokens are subword units — approximately 1 token per 4 characters of English text.
BPE produces efficient vocabularies for common words and splits rare strings into more tokens.
Non-English text and code often use more tokens per character than English prose.
API cost scales linearly with tokens — output tokens cost more than input tokens.
Use count_tokens() to estimate cost before running long agent pipelines.

Counting Tokens Before Running Agents

Always estimate token usage before running expensive agent pipelines on large inputs. OpenClaw’s token counter uses the same tokeniser as the target model:

from openclaw.utils import count_tokens

system_prompt = "You are a helpful assistant..."
user_input = open("large_document.txt").read()

total = count_tokens(system_prompt + user_input, model="gpt-4o")
cost_estimate = (total / 1_000_000) * 2.50  # $2.50 per 1M input tokens
print(f"Estimated input cost: ${cost_estimate:.4f} for {total:,} tokens")

if total > 100_000:
    print("Warning: input exceeds 100K tokens, consider chunking.")

Practical Token Reduction Techniques

Chunk large documents. Split documents into 4K–8K token chunks and process only the relevant chunks (retrieved via RAG) rather than sending the full document.
Compress tool outputs. Tool responses are often verbose JSON. Strip unused fields before returning them to the agent context.
Use concise system prompts. A 2,000-token system prompt adds 2,000 tokens to every single agent call. Review and trim system prompts regularly.
Cache repeated inputs. Use semantic caching for prompts that are identical or near-identical. The cached response costs zero tokens on subsequent calls.