Why AI benchmarks are more uncertain than they look

The question universe is the theoretical sum of all possible questions that could represent a specific skill, such as physics, law, or coding. Current AI benchmarks like MMLU or MATH only use a small sample of these questions. Anthropic’s research suggests that a model's score should not be viewed as an absolute truth, but rather as an estimate of its performance across this entire unseen super-population. Without acknowledging this "universe," researchers may mistake a model's luck on a specific set of questions for actual underlying mastery of a subject.

Standard statistical math often assumes every question is an independent event, but many evaluations use "clustering," where multiple questions are tied to a single long passage. If a model misunderstands a specific passage, it will likely miss all related questions, meaning the questions are not independent draws. Ignoring this clustering can result in standard errors that are three times smaller than they should be, giving researchers a false sense of confidence in results that might actually be statistical noise.

Instead of forcing a model to pick a single answer (like "A" or "B"), researchers can look at the internal probability the model assigns to the correct token. For example, if a model assigns a 72% probability to the correct answer, it receives a score of 0.72. This method eliminates the randomness associated with token generation and "temperature" settings. It provides a more nuanced, continuous score that can reduce measurement variance by up to two-thirds compared to traditional pass/fail grading.

A paired-difference analysis compares two models by looking at how they performed on the exact same questions, rather than just comparing their final average scores. Since frontier models often struggle with or excel at the same specific questions, their results are highly correlated. By focusing on the "gap" per question, researchers can subtract out the noise caused by question difficulty. This makes the measurement of the difference between two models much more precise and can even reveal that a model with a lower average score is actually the statistically significant winner.

Power Analysis is a mathematical formula used to determine if an evaluation is sensitive enough to detect a real difference between models before the test is even run. It helps researchers calculate the necessary sample size—often requiring at least a thousand independent questions—to ensure a result isn't just a false positive. This prevents researchers from "weighing a diamond on a bathroom scale" by ensuring the test has enough statistical power to see small performance gains, such as a 2% or 3% improvement.