Token
Also called tokens, tokenization, tokenizer
A small chunk of text, often part of a word, that AI models read and write. Usage and prices are counted in tokens.
A token is the basic unit of text for a language model. Before a model reads your prompt, a tokenizer splits it into tokens: common words are often one token, longer or rarer words become several, and spaces and punctuation are included. In English, one token averages about four characters, so 1,000 tokens is roughly 750 words.
Tokens matter for three practical reasons. Context windows are measured in tokens, so they set how much text fits. API prices are charged per token, usually with output tokens costing more than input tokens. And speed is often measured in tokens per second.
Different models use different tokenizers, so the same text can be a different number of tokens in each. Languages other than English, code, and text with lots of numbers or unusual symbols often use more tokens per word. Tokenization also explains some odd model weaknesses, such as miscounting letters in a word, because the model sees chunks rather than individual letters.
Example
The sentence "Tokenization is surprisingly important." might split into tokens like "Token", "ization", " is", " surprisingly", " important", "." That is 6 tokens for 4 words.
Common mistake
Estimating costs from word counts alone. A prompt that includes a long system prompt, retrieved documents and conversation history can be ten times larger than the visible user message. Measure real token counts from the API's usage data, and budget for both input and output.