If you are building applications on top of AI APIs, understanding tokenization is not optional — it directly affects how much you pay, what your model can process, and how reliably your application behaves. This guide explains tokenization from first principles, covers the differences between major model families, and gives you concrete techniques to work with tokens effectively.
A token is the basic unit of text that a language model processes. Tokens are not words, characters, or sentences — they are chunks of text produced by a statistical compression algorithm called Byte Pair Encoding (BPE). In English text, one token corresponds to roughly 3–4 characters on average, or about three-quarters of a word.
Here is how the phrase "tokenization matters" might be split into tokens by a typical BPE tokenizer:
Note that "tokenization" is split into two tokens, while " matters" (with its leading space) becomes a single token. This is typical BPE behavior — common words and word fragments become single tokens, while unusual or long words get split.
BPE is a compression algorithm originally designed for data compression, adapted for natural language processing. The training process works like this:
The result is a vocabulary where common English words, common word fragments, programming language keywords, and common multi-word phrases are represented as single tokens. Less common text — unusual words, rare names, foreign languages, or highly technical jargon — gets split into more tokens.
Practical implication: Common English prose is tokenized efficiently (roughly 1 token per 4 characters). Code with lots of variable names, strings, and special characters tokenizes less efficiently. Non-Latin scripts (Chinese, Japanese, Arabic) can require 2–4× more tokens per character than English.
Every major AI API provider bills by the number of tokens processed — separately for input tokens (what you send) and output tokens (what the model generates). Input tokens include your system prompt, conversation history, any documents or context you include, and the current user message. Output tokens are everything the model writes in response.
Output tokens are significantly more expensive because text generation requires autoregressive computation — the model must run a forward pass for every single output token, whereas input tokens are processed in a single parallel pass. This is why output tokens typically cost 4–10× more than input tokens depending on the model.
At scale, this asymmetry has major implications for application design. If you can instruct the model to produce shorter responses without losing information, the savings are dramatic. A response that's 500 tokens instead of 800 tokens saves 37.5% of your output costs — and output is where most of your budget typically goes.
OpenAI's GPT-5.6 family and their predecessors use the cl100k_base tokenizer, which has a vocabulary of approximately 100,000 tokens. This tokenizer is optimized for English and code, and handles most European languages well. It includes special tokens for common programming constructs (curly braces, semicolons, import statements) which means code is typically tokenized efficiently.
Claude models use a tokenizer closely based on cl100k_base. Token counts for the same English text will be extremely similar to OpenAI's tokenizer — usually within 1–3%. The main differences appear in how edge cases and special characters are handled. For most practical purposes, if you have a token count for a prompt from a GPT model, you can use it as a reliable estimate for Claude too.
Gemini models use a SentencePiece tokenizer with a different vocabulary from cl100k. For standard English prose, token counts are in a similar ballpark, but the tokenization strategy differs. Gemini's tokenizer was trained on a broader multilingual corpus, which means it handles non-English languages more efficiently than BPE-based tokenizers. However, for some types of code or technical content, you may see slightly different counts.
Understanding how different content types tokenize helps you estimate and optimize costs:
Every model has a maximum context window — the total number of tokens it can process in a single API call. This limit applies to input plus output combined. If your input is 190,000 tokens and the model has a 200,000 token context window, you only have 10,000 tokens available for the model's response.
In 2026, context windows have grown dramatically. GPT-5.6 models support 270,000 tokens; all Claude 5 and Haiku 4.5 models support 200,000 tokens; all Gemini models support 1,000,000 tokens (one million). A 1M token context window can hold:
Despite these large context windows, cost increases linearly with context size. Sending 500,000 tokens of context costs the same per-token rate as sending 1,000 tokens. Large context does not cost more per token — but using a large context costs more in absolute terms because you are processing more tokens.
Use a token counter before making API calls with large inputs. A rough count helps you avoid hitting context limits and lets you budget accurately. The rule of thumb: 1 token ≈ 4 English characters, or roughly 1,333 tokens per 1,000 words. Use our token counter for more accurate estimates across specific model families.
Every API response includes usage metadata with the exact input and output token counts. Log these values for every request. After a few days of production traffic, you will have accurate data on your average token consumption per endpoint, which is essential for cost forecasting and optimization.
For applications that process large documents, consider these strategies: extract and send only the relevant sections rather than full documents; summarize background context that doesn't change frequently; use retrieval-augmented generation (RAG) to fetch relevant chunks rather than stuffing entire knowledge bases into the prompt.
Different models use different tokenizer vocabularies trained on different corpora with different vocabulary sizes. OpenAI's cl100k_base vocabulary has ~100,000 tokens. Anthropic's tokenizer is closely related. Google's SentencePiece tokenizer is trained differently. A vocabulary that includes a common word as a single token will count it as 1 token; a vocabulary without that word will split it into fragments totaling 2–4 tokens. The differences are usually small for common English text but can be significant for specialized content, code, or non-English languages.
For OpenAI models, use the open-source tiktoken library (Python). It gives exact token counts using the actual tokenizer. For Claude, Anthropic's API includes a count_tokens endpoint (and returns usage in API responses). For Gemini, use the countTokens method in the Gemini API. Browser-based token counters like ours use a BPE approximation that is accurate within ±5% for typical English text — sufficient for estimation and optimization work, but not for applications where you need to enforce precise context limits.
Multimodal inputs (images, audio, video) are converted to token equivalents for billing purposes, but the conversion formula varies by provider and model. OpenAI prices images based on their resolution, with low-detail images costing ~85 tokens and high-detail images costing 170 tokens per 512×512 tile. Anthropic charges based on image dimensions. When building applications that process images, factor image token costs into your per-request budget calculations — a single high-resolution image can add hundreds to thousands of tokens worth of cost.
The API returns an error (typically 400 or 422) indicating the context length was exceeded. In applications, this means you need to either truncate the input, summarize it, or split it into smaller chunks. It is always better to check token count before submitting rather than handling errors in production. For agentic applications that accumulate conversation history, implement a context management strategy from the beginning — either a sliding window, periodic summarization, or conversation reset.
Efficiently: common English prose, popular programming languages (Python, JavaScript, Java), HTML with standard tags. The tokenizer vocabulary was built from real-world web text and code, so common patterns are well-covered. Inefficiently: base64-encoded data, encrypted or hashed strings, URLs with long random parameters, highly repetitive text with unusual characters, and languages with complex scripts (Thai, Khmer, some Arabic dialects). If you are processing unusual content types, measure the actual token count rather than relying on the 4-characters-per-token rule of thumb.