Why Benchmarks Lie

The numbers are extraordinary. Large language models now pass the Uniform Bar Examination, score in the top percentiles on medical licensing tests, solve competition-level mathematics problems, and achieve human-level performance on graduate-level science questions. Each new model release brings a fresh cascade of benchmark scores, each one higher than the last, each one accompanied by the implicit narrative: AI is getting smarter. We are on an exponential curve. General intelligence is within reach.^[1]

The benchmarks are not fabricated. The performance is real. When a model achieves a 90th-percentile score on the bar exam, it has genuinely produced responses that professional examiners would score at that level. When it solves mathematical olympiad problems, the proofs check out. When it passes the United States Medical Licensing Examination, its diagnostic reasoning is often clinically sound.^[2] These are not tricks or statistical artefacts. They represent a genuine and remarkable expansion of what automated systems can do.

The policy implications seem to follow naturally. If AI can pass the bar, it can assist with legal work. If it can pass medical boards, it can support clinical decision-making. If it can solve graduate-level science problems, it can accelerate research. The benchmarks serve as a shared legibility layer — a way for researchers, investors, policymakers, and the public to track progress along a seemingly clear trajectory from narrow capability to general intelligence.

This narrative is compelling, internally consistent, and almost entirely misleading.

The Map Is Not the Territory

The problem begins with a question that is almost never asked: what, exactly, does a benchmark score measure? The assumption is that it measures intelligence, or capability, or competence — some general property that transfers from the test to the real world. But this assumption is doing a great deal of work, and it collapses under scrutiny.

A benchmark is a standardised evaluation conducted under controlled conditions. The questions are drawn from a fixed distribution. The format is constrained — typically multiple choice or short-form response. The context is complete and self-contained; the test-taker is not required to gather additional information, manage ambiguity, or navigate the social and institutional complexities that surround real-world decision-making. These constraints are not incidental. They are what make benchmarks tractable, reproducible, and comparable. But they are also what make benchmarks easy in a way that reality is not.^[3]

François Chollet articulated this distinction with characteristic precision. In “On the Measure of Intelligence,” he argued that most AI benchmarks confuse skill with intelligence.^[4] Skill is performance on a specific task distribution. Intelligence is the efficiency with which an agent acquires new skills when confronted with novel tasks — tasks it has not been optimised for. A model that memorises the bar exam corpus and achieves a high score has demonstrated skill. Whether it has demonstrated intelligence depends on whether it can transfer that capability to genuinely novel legal reasoning under conditions that differ from the exam format.

The evidence suggests that transfer is far more limited than benchmark scores imply. Models that excel on standardised tests often struggle with slight reformulations of the same problems — changes in wording, novel framings, or problems that require integrating information across domains in ways the training data did not exemplify.^[5] This is the signature of sophisticated pattern matching, not general reasoning. A medical student who passes board exams but cannot diagnose an unusual presentation would be considered undertrained. We hold AI to a different standard, and the benchmarks enable this double standard.

When the Measure Becomes the Target

The British economist Charles Goodhart observed in 1975 that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.”^[6] Marilyn Strathern later generalised this: “When a measure becomes a target, it ceases to be a good measure.”^[7] This principle, known as Goodhart's Law, is one of the most robust findings in social science, and it applies to AI benchmarks with particular force.

The AI industry optimises for benchmarks. This is not conspiratorial — it is structural. Researchers are evaluated on benchmark improvements. Companies market their models based on benchmark scores. Funding decisions are influenced by benchmark performance. The entire ecosystem is calibrated to the signal that benchmarks provide. And Goodhart's Law predicts exactly what happens next: the benchmarks become the target, and the correlation between benchmark performance and real-world capability degrades.^[6]

The mechanisms are well-documented. Training data contamination — where benchmark questions or closely related material appear in the training corpus — is increasingly difficult to prevent as models are trained on ever-larger slices of the internet.^[8] Benchmark overfitting, where architectural and hyperparameter choices are tuned to specific benchmarks rather than general capability, is pervasive. And there is a subtler effect: the benchmarks themselves shape what capabilities are developed. Problems that are easy to benchmark get attention and resources. Problems that are hard to benchmark — common-sense reasoning, creativity, robustness to distribution shift, ethical judgement — are systematically under-invested.^[3]

Raji et al. named this phenomenon directly in their analysis of AI evaluation practices, arguing that the field suffers from a “benchmark culture” that privileges quantitative comparisons over meaningful assessment of system behaviour.^[3] Their paper, with its deliberately provocative title referencing “the everything in the whole wide world benchmark,” makes the case that the proliferation of benchmarks has not improved our ability to evaluate AI systems — it has given us a false sense of precision about questions that remain fundamentally open.

The ARC-AGI benchmark represents a deliberate attempt to break this pattern.^[9] Designed by Chollet to test novel abstraction rather than memorised patterns, it generates tasks that are trivially easy for humans but remain challenging for large language models. The benchmark explicitly targets the gap between pattern recall and genuine generalisation. The persistent difficulty that state-of-the-art models face on ARC-AGI tasks — despite their dominance on conventional benchmarks — is perhaps the clearest evidence that benchmark success and cognitive capability are measuring different things.

The Deeper Problem: What We Fail to Measure

The limitations of benchmarks become starkest when you consider what they do not measure at all. Intelligence — whether human or artificial — is not a single scalar quantity. It encompasses perception, reasoning, planning, learning, social cognition, emotional regulation, metacognition, and the ability to operate effectively under uncertainty and resource constraints. Benchmarks, by their nature, isolate individual capabilities and test them in sterile conditions. They cannot capture the integration of capabilities that characterises intelligent behaviour in complex environments.^[4]

Consider metacognition — the capacity to monitor one's own reasoning processes, recognise uncertainty, and adjust strategy accordingly. A human expert, when confronted with a problem at the edge of their competence, typically knows they are uncertain. They slow down, seek additional information, consult colleagues, or explicitly flag their uncertainty. This metacognitive awareness is not a luxury; it is what makes expertise safe. A confident expert who cannot recognise the limits of their knowledge is more dangerous than a cautious one who can.^[10]

No current benchmark measures this. Models are evaluated on the correctness of their outputs, not on the quality of their uncertainty estimation. A model that answers every question with supreme confidence — right or wrong — scores exactly the same as a model that correctly identifies when it is uncertain, as long as their accuracy rates are equal. The benchmark is blind to a dimension of capability that may matter more than raw accuracy in real-world deployment.

Similarly, benchmarks do not measure robustness to adversarial or out-of-distribution inputs. They do not measure the ability to ask clarifying questions when a problem is underspecified. They do not measure the ability to explain reasoning in a way that allows human oversight. They do not measure the ability to integrate new information that contradicts prior beliefs. They do not measure the ability to operate safely under resource constraints. Each of these capabilities is essential for trustworthy AI deployment, and none of them appears on any leaderboard.^[11]

Evaluating the Process, Not Just the Output

If benchmarks measure the wrong thing, what would measuring the right thing look like? The answer points toward a fundamental shift in evaluation philosophy: from output-oriented assessment to process-oriented assessment.^[4]

In human education, we already understand this distinction. A mathematics exam that only checks final answers tells you whether the student got the right number. A mathematics exam that requires showing work tells you whether the student understands the underlying principles. The second form of evaluation is harder to administer, harder to grade, and far more informative. It distinguishes between a student who understands calculus and a student who memorised the answer key.

The analogy to AI evaluation is direct. Current benchmarks check the “final answer” — did the model produce the correct output? Process-oriented evaluation would examine the reasoning chain: Did the model identify the relevant information? Did it consider alternative hypotheses? Did it flag its uncertainties? Did it reach its conclusion through logically valid steps? Are those steps independently verifiable?^[12]

This shift requires systems that can externalise their reasoning in a form that admits verification. Chain-of-thought prompting is a step in this direction, but it is a weak one — the model generates a reasoning trace, but that trace is itself generated by the same probabilistic process and may be a post-hoc rationalisation rather than a faithful record of the computational steps that produced the output.^[13] What is needed is not a model that can explain its reasoning, but a system architecture where the reasoning steps are first-class objects that can be inspected, validated, and challenged.

Imagine a system where every claim in a generated response is linked to a reasoning step, every reasoning step is linked to evidence, and every evidential link can be independently checked. The evaluation of such a system would not ask “Did it get the right answer?” but “Is every step in its reasoning chain valid, and does the chain support the conclusion?” This is a much harder criterion to satisfy, and a much more meaningful one.

Toward Honest Measurement

The practical implications extend beyond evaluation methodology. If we take seriously the claim that benchmarks are unreliable proxies for real-world capability, then the entire framework for assessing AI readiness needs to change. Deployment decisions based on benchmark scores are, at best, weakly informed. At worst, they create a false sense of security that leads organisations to deploy systems in high-stakes contexts where the gap between benchmark performance and real-world robustness causes genuine harm.^[5]

What would better evaluation look like in practice? Several principles suggest themselves. First, evaluation should be adversarial by default — not just testing whether the system can answer standard questions correctly, but actively probing for failure modes, edge cases, and brittle behaviours. Red-teaming, which subjects systems to deliberate attack, is more informative than passive benchmarking.^[11]

Second, evaluation should measure calibration — the correspondence between the system's expressed confidence and its actual accuracy. A system that says “I am 90% sure” and is right 90% of the time is well-calibrated. A system that says “I am 90% sure” and is right 60% of the time is dangerously overconfident. Current benchmarks do not measure calibration, which means they cannot distinguish between a knowledgeable system and a confidently wrong one.^[10]

Third, evaluation should be longitudinal — testing not just point-in-time performance but consistency over extended interactions, resistance to drift, and the ability to maintain coherent reasoning across complex, multi-step tasks. A system that performs brilliantly on isolated queries but degrades over a thirty-step reasoning chain is not ready for the real world, no matter what its benchmark scores say.

Fourth, and perhaps most importantly, evaluation should be process-transparent. Systems should be required to expose their reasoning chains in a format that admits independent verification. Not as a nice-to-have feature, but as a precondition for deployment in any context where errors have consequences. If you cannot audit the reasoning, you cannot trust the conclusion.^[12]

The Score Is Not the Capability

None of this is to say that benchmarks are useless. They serve a real purpose as coarse-grained indicators of progress, as debugging tools for identifying capability regressions, and as shared reference points for research communication. The problem is not that benchmarks exist but that they are treated as definitive measures of intelligence rather than partial, imperfect, gameable proxies for specific capabilities under specific conditions.^[3]

The history of psychometrics offers a cautionary parallel. IQ tests, when first developed, were treated as objective measures of general intelligence. Decades of research revealed them to be culturally biased, narrowly focused, and weakly predictive of real-world outcomes beyond academic performance. The tests themselves were not worthless — they measured something real — but the claims made on the basis of those measurements far exceeded what the data could support. AI benchmarks are in a similar position today: real measurements, inflated interpretations.^[4]

The path forward requires intellectual honesty. When a model achieves a high benchmark score, the correct interpretation is not “this model is intelligent” but “this model performs well on this specific evaluation under these specific conditions.” The gap between those two statements is where most of the interesting — and most of the dangerous — questions live.

What would it mean to build AI systems that could survive genuinely rigorous evaluation? Systems whose reasoning chains are transparent and independently verifiable. Systems that know what they do not know and say so. Systems that perform consistently under adversarial conditions, not just friendly ones. Systems that can be audited, challenged, and held accountable.

The benchmarks will continue to rise. The scores will continue to impress. The question that matters is whether we have the discipline to look past the numbers and ask what they actually mean — and the honesty to build evaluation systems worthy of the capabilities we claim to measure.