研究者究竟怎样衡量 AI 的智能？How Exactly Do Researchers Measure AI Intelligence?

Models are said to pass the Bar Exam and turn out spotless code — but on what basis do we call them "smart"? Go inside the lab and see the benchmarks, Turing tests and adversarial evals that researchers use in 2026 to put a number on machine intelligence.

Another launch event, another round of talk about "unprecedented intelligence" — that is the rhythm of this field. The pitch usually includes 90th percentile on the SATs, impeccable Python, a pass on the medical licensing exam. A sensible user is entitled to push back: on what evidence are such claims made, and is anything actually being thought? Or is this simply very skilled test-taking?

Few problems in computer science are messier. Human IQ testing has had a century of refinement behind it; AI evaluation has no such stability — the target keeps moving. What follows is a look behind the curtain at the places where models get graded: the human "vibe checks", the adversarial attacks, and the benchmark suites that together decide whether a system deserves to be called smart.

01The Hard Part: Defining What "Smart" Means for a Machine

Any test presupposes a definition, and defining intelligence for a machine drags you straight into philosophy. Memorising an entire encyclopedia — does that count as smart? Turning out a lovely poem while failing to grasp that 2+2=4 — is that intelligent?

The workaround is to split the concept apart, domain by domain:

• Knowledge Retrieval: how reliably does it pull up facts from history, science and law?
• Logical Reasoning: how does it handle maths that unfolds over several steps, or draw a conclusion from stated premises?
• Coding & Spatial Awareness: does it produce software that runs, and does it grasp physical relationships?
• Linguistic Nuance: how well does it handle sarcasm, idioms and complicated instructions?

Broken into categories like these, intelligence becomes testable domain by domain. What emerges is not one "IQ score" for the machine but something closer to a full report card spanning many subjects.

02The Turing Test, and Why Researchers Moved On

Raise the subject of AI testing and the Turing Test will come up before long. Alan Turing put it forward in 1950, and the setup is minimal: a judge exchanges text with one person and one machine; if the judge cannot tell them apart with confidence, the machine has passed.

Its standing nowadays is that of a historical landmark rather than a working instrument. Genuinely measuring machine intelligence called for something stricter — something you can quantify and that resists being "gamed".

03Benchmarks: Today's Actual Intelligence Exams

This is where benchmarks took over. They are enormous standardized datasets, thousands of questions spanning many disciplines, and a freshly trained model is run through them to produce a score. The ones that carry most weight in 2026:

Reading the Report Card

Evaluation never rests on a single number. The picture is a radar chart plotting performance across the whole set. A model at 95% on MMLU (knowledge) yet 40% on ARC (reasoning) tells researchers it has absorbed a great deal of schooling while falling short on critical thinking.

04Red Teaming: Probing Intelligence by Breaking It

A strong maths score says nothing about safety or genuine understanding, which is the gap red teaming fills. Experts are brought in — friendly hackers, linguists, domain specialists — and paid to "break" the system deliberately, or to lure it into exposing where its limits lie.

A model that a commonplace logical fallacy can trip up will be marked down for weak reasoning, however lofty its benchmark numbers happen to be.

05Human Evaluation, a.k.a. the Vibe Check

Numbers never capture everything. A model can crack a difficult physics equation and then explain it in prose that is baffling, mechanical or faintly patronising. Human Preference Evaluations exist to catch exactly that.

Concretely: a rater sees two replies, produced by two different models for one prompt, and picks the one that strikes them as more helpful, more honest and less harmful. Those verdicts then feed back into training — the technique known as RLHF — teaching the system what people actually prize in an answer they would call smart.

06Where AI Testing Breaks Down: Goodhart's Law

Statistics has a well-known adage for this, called Goodhart's Law: once a metric is treated as the goal, it stops working as a metric. Nothing troubles AI evaluation more today.

Since labs compete for top placement on suites such as MMLU, the incentive to tune for those exact tests is enormous — and it produces two familiar problems:

• Data Contamination: the model "memorises" benchmark questions inadvertently while training on a vast web crawl. That 90% reflects prior exposure to the exam, not intelligence.
• Benchmark Hacking: developers can make small adjustments that suit a given benchmark's exact format, while the reasoning underneath stays exactly as weak as before.

The countermeasure is the move to Dynamic Evals: sets that shift constantly, get refreshed every week and stay under strict secrecy. If you want to track where evaluation methods are heading, this week in AI research will flag the arrival of new benchmarks that have not been contaminated.

07Questions Readers Ask Most