AI 是如何决定接下来该说什么的？How Does AI Choose What to Say Next?

It looks like magic; underneath it is arithmetic. Here is the full chain — from cutting your sentence into tokens to sampling one candidate out of thousands — that lets a large language model produce human-sounding text one step at a time.

Ask ChatGPT or Claude a question and an answer starts appearing within a second or two: fluent, on-topic, seemingly considered. It is tempting to read intent into it, as though something behind the screen were weighing up a reply. Nothing of the sort is happening. No memory is being retrieved and no opinion is being formed; what you are watching is an extremely elaborate guessing game.

Strip it down and the whole question is this: given every word written so far, which word has the best odds of coming next? The model asks that over and over, dozens of times a second, and each answer lengthens the sentence by a few characters. Below we take the machinery apart — tokens, probability rankings, the "attention" trick that keeps long passages coherent, and the setting that decides whether the output comes out dry or playful.

01The Core Idea: Autocomplete, Scaled Up

The handiest comparison is the suggestion strip above your phone keyboard. Type "I'll see you" and it offers "tomorrow" or "later" — not because it grasps your plans, but because that pairing has turned up millions of times in messages you wrote before.

A large language model is that same mechanism turned up to an absurd volume. Swap your personal message history for a large slice of the public web — books, news, forum threads, source code — and the pattern table grows vast enough to cover almost any phrasing in English and plenty of other languages. So there is no lookup happening when you ask something. The reply gets assembled word by word, each step betting on the most sensible continuation given everything the model has previously read.

02Phase 1: Why Words Have to Become Numbers

A machine has no idea what "apple" or "run" refers to. Numbers are all it can work with. So before any prediction can happen, your text has to be re-encoded into something arithmetic-friendly — a step called tokenization.

Tokens rarely line up neatly with words. A token might be a syllable, a common prefix, or a lone character. "Unbelievable", for instance, often splits into three: "un", "believe", "able". Every token carries its own ID number, and the model multiplies those numbers through elaborate matrices to work out how they relate. For the deeper version of this step, how text gets turned into arithmetic walks through how text turns into arithmetic.

03Phase 2: Ranking Every Candidate

With your prompt processed, the model consults the parameters it accumulated during training and produces a long list of every word that could plausibly follow, each tagged with a likelihood score. That list is the probability distribution.

Read the chart like this: if the preceding words were "She opened the", the model scores "door" far above everything else. Yet it does not simply take the top entry every time — do that and the text loops and bores. Sampling draws from the small set of leading candidates instead, and that is where variety and a little personality enter the output.

04Phase 3: How Context Steers the Choice

Context was the weak point of early models. Feed one a sentence beginning "The bank was steep" and it might read "bank" as a financial institution. The fix arrived with an architecture called the Transformer, whose "self-attention" mechanism lets a model weigh every other word in the sentence at the same time before settling on a meaning.

That is what separates a riverbank from a savings bank. Words such as "steep" or "money" pull the probabilities in different directions, and the model follows the pull. Holding context across long stretches is what makes the output feel coherent; the architecture explained in depth covers the architecture in more detail.

05Temperature: The Dial Between Safe and Surprising

You may have noticed the same model sounding flat and factual one moment, then loose and inventive the next. A setting called temperature is usually what changed. Think of it as a dial governing how much risk the model may take when picking words.

• Low temperature (0.2): the model stays cautious, nearly always taking the highest-probability token. Good for code and factual summaries, where being right matters more than being interesting.
• High temperature (0.8 and up): it starts gambling, occasionally choosing a lower-ranked word because it fits the mood. The prose gets livelier and more human — and the odds of outright nonsense climb with it.

06Why Models Make Things Up

"Hallucination" — the confident delivery of something flatly untrue — is the most common complaint about AI, and it traces straight back to how the next word gets chosen. What the model optimises for is plausibility, not accuracy.

A fabricated fact can slide through whenever it happens to sit in a familiar sentence shape; sounding right is the only test applied. There is no verification module anywhere in the loop, just a probability calculator. Which is why anything consequential deserves a check against a source. It is also the line between AI and conventional automation: rules drive one, patterns drive the other — where automation ends and AI begins explains the split.

07Where the Skill Comes From: Training Data

A model can only be as capable as the material it learned from. Show it a domain it has never encountered — a niche engineering manual, say — and it has no basis for guessing the next word in that register. Hence the enormous spending on assembling broad, high-quality corpora: the more good examples a model absorbs, the sharper its probability estimates get on narrow subjects. why training sets have to be enormous goes into the detail.

08Seen in Practice: Translation

Translation is where this mechanism looks most impressive. Older tools worked word by word and produced stilted, wooden output. A modern model instead reads the whole source sentence for context, then generates the target-language wording with the highest odds of carrying the same meaning.

It is not deciding how to render a bare "hello" in isolation. Tone, formality and cultural setting all feed in, and the next word shifts accordingly — a stiff greeting in a formal setting, something offhand among friends. That sensitivity to context is what separates current translation from the earlier generation; the pipeline behind machine translation shows the process step by step.

09Common Questions