AI 中的 tokenization 是什么？What Is Tokenization in AI?

Send a message to an AI and its opening move is always the same: slice your words into fragments. That opening move, tokenization, sits beneath everything ChatGPT, Claude, and Gemini produce. Below we unpack what it really means, why it carries weight, and the difference it makes to how you work with AI.

You may have spotted an odd habit in AI. Ask it to tally the letters within a word and the answer sometimes misses; ask it to flip a sentence backwards and it falters. These aren't marks of dimness; they follow directly from a step named tokenization. Grasp what a token is and the model's oddities suddenly fall into place.

Tokenization belongs among the rare concepts lying silently beneath every AI exchange you have ever had, yet almost nobody explains it in everyday words. That gap closes here. By the end, you will know precisely how your words are handled, why token ceilings shape your chats, and how that knowledge lets you use AI to better effect.

01The Plain-English Meaning

Begin at the very bottom. Computers cannot read text as people do. The word hello reaches your brain as one familiar whole; a computer encounters nothing but a chain of characters, h, e, l, l, o. To perform anything clever with language, an AI model first needs a route out of human text and into a form it can treat mathematically, and that route is precisely tokenization.

At its simplest, tokenization is the act of cutting text into fragments named tokens. Picture a sentence pulled apart into LEGO bricks; each brick is one token. The AI lifts these bricks, inspects them, works out how they connect, and draws on that grasp to assemble a reply, brick by brick.

This ties directly to the broader way AI models treat language. For the complete picture of what follows once tokens exist, our guide on how AI grasps human language traces the whole route from your first keystroke to the finished reply.

02Why Tokenization Genuinely Matters to You

You may well be asking why the internal slicing of your text should concern you. The reason is that tokenization carries three very practical, everyday consequences, each touching your daily experience of AI tools.

1. Context Windows: Why an AI Loses the Thread in Long Chats

Every AI model holds a maximum number of tokens at once, a ceiling named the context window. Treat it as the model's working memory. Once a chat outgrows that ceiling, the model begins discarding earlier portions to free space for fresh material.

That is why, deep into a lengthy chat, an AI can seem to lose something you stated early on. Nothing is truly forgotten; that material simply fell outside the context window. Our detailed explainer on what an AI model's context window really means shows exactly how this works.

2. API Costs: The Token as the Billing Unit

When you reach AI through an API, the way developers build apps, each token is metered, both the tokens you send as your prompt and those the model returns as its reply. Knowing the token counts helps you draft leaner prompts and forecast expense reliably. A token cost calculator sits further down this very page for precisely that purpose.

3. Accuracy on Letter-Level Jobs

Here is a fact that catches most people out: AI is surprisingly weak at counting letters within words. Put to ChatGPT how many R's sit inside strawberry and the reply can miss. Tokenization is the direct cause. The model never meets individual letters; it meets tokens. Treat strawberry as one token and the model has no route to inspect the characters unless expressly built for it. Once that is clear, these silly slips stop surprising you.

03How Tokenization Actually Works

Here is the sequence, step by step, from the instant you send an AI a message:

1. 1

   Pre-processing: tidying the text

   1 First comes a light tidying: punctuation is dealt with, special characters are accounted for, and the text is normalised, so processing stays consistent however you happened to type.

2. 2

   Cutting into tokens

   2 The tokenizer cuts your text against a learned vocabulary, commonly 50,000 to 100,000 possible tokens. Frequent words such as the, is, and you each take a single token, while rarer words and technical terms divide into sub-word pieces. The word tokenization itself may split into two tokens, token and ization.

3. 3

   Token IDs: attaching numbers

   3 Every token is then swapped for a unique integer, its token ID. Thus hello might become 15496 and world might become 995. From here the model never meets words at all; only these numbers ever reach it.

4. 4

   Embeddings: turning numbers into meaning

   4 Each token ID next becomes a dense numerical vector named an embedding, a long list of numbers encoding the token's meaning and its ties to other tokens. These vectors are what the transformer model actually treats. To go deeper, see our piece on what occurs inside a neural network.

5. 5

   Output: producing tokens in sequence

   5 When a reply is generated, the model never writes a full sentence at once. It forecasts the likeliest next token, folds it into the context, forecasts the one after that, and so on until the reply is complete. That is why you notice a gentle streaming effect as the answer appears.

04Try It: A Live Token Counter

Type any sentence underneath and watch it divide into tokens immediately. It builds an intuitive sense of how the AI actually sees your words.

05What Is BPE? The Algorithm Driving Today's Tokenization

Most leading AI models, GPT-4, Claude, and Gemini among them, rely on a tokenization technique named Byte Pair Encoding (BPE). The name sounds daunting, yet the notion is elegantly plain.

BPE begins with each single character as its own token, so hello opens as five tokens, h, e, l, l, o. It then scans the training data for the commonest token pair; should t and h frequently sit together, they merge into one token, th. The next commonest pair then merges, and the next, repeating thousands of times until the vocabulary reaches its target size, usually about 50,000 to 100,000 tokens.

The outcome is a vocabulary in which frequent English words each take a single token, while rare words, technical jargon, and words from less common languages divide into smaller sub-word pieces. This explains the following:

This also clarifies a counterintuitive quirk: inserting spaces ahead of words or using unusual capitalisation can shift the token count. The tokenizer reacts to these details in ways that often surprise newcomers to AI.

Token Cost Calculator

When you reach AI through an API, every token carries a price. Use this calculator to forecast what your prompts cost across various models:

06Languages Differ, and Token Counts Differ Wildly

Among tokenization's most important yet least discussed traits is how sharply it varies by language. The vocabularies of most leading AI models lean heavily toward English, since English made up the bulk of their training data. That carries genuine everyday consequences.

The same thought expressed in English may take 10 tokens, while Arabic, Hindi, or Thai may need 20-30 tokens to say the identical thing. The cause is a thinner set of pre-built tokens for those languages, forcing the tokenizer more often to break words into smaller, character-level fragments.

This remains an active front for improvement across the AI field. Newer models increasingly train on richer multilingual data, which helps their tokenizers grow more efficient across languages. For now, though, anyone building AI apps for a non-English audience should weigh token efficiency as a serious cost and performance factor.

07Common Myths About AI Tokens, Cleared Up

A good deal of confusion surrounds tokens in everyday use. Let us settle the most widespread misunderstandings:

Grasping tokenization also clarifies why AI-made images behave so differently from text. Images are not tokenized like words; they use visual patches or latent-space representations instead. If that side intrigues you, our article on how AI turns text into images gives the full picture.

08Test Yourself: A Token Knowledge Quiz

Ready to gauge what you have picked up? Three questions with immediate feedback.

09Frequently Asked Questions