Tokens are small pieces of text—words, parts of words, or punctuation—that AI models process instead of raw text. Tokenization splits your input into these pieces so the model can read, generate, and price its output accordingly.
Why models use tokens instead of words
Language models don't understand letters or full words directly; they work with numerical representations. Breaking text into tokens lets a model handle rare words, typos, and multiple languages by combining smaller, reusable pieces, similar to how embeddings turn tokens into vectors the model can compute with.
How tokenization affects cost and speed
Most LLM pricing is based on token counts for both input and output, so longer prompts and responses cost more and take longer to generate. A rough rule: one token is about four characters in English, so 100 tokens is roughly 75 words.
Where operators actually notice this
If you're building AI features into a product, token limits define how much context a model can hold at once, which shapes everything from chatbot memory to document summarization. Understanding tokenization helps you estimate costs, debug odd model outputs, and set realistic limits with context windows.
Frequently asked
Is a token the same as a word?
Not exactly. Common words are often one token, but longer or rare words get split into multiple subword tokens.
Why do token counts matter for cost?
Most AI providers charge per token processed, so more tokens in your prompt or response means a higher bill.
Do different AI models tokenize text the same way?
No, each model family typically uses its own tokenizer, so the same text can produce different token counts across models.