Why AI Benchmarks Are Less Accurate Than They Look

Nia: You know, Eli, I was looking at some AI leaderboards this morning, and it hit me—how do we actually know if one model is truly "smarter" than another, or if it just got lucky with the questions?

Eli: That is exactly the question Anthropic tackled in their research late last year. It turns out, "luck of the draw" is a huge factor. They found that when you account for how questions are grouped, the actual margin of error can be over three times larger than what researchers usually report.

Nia: Wait, three times? That’s a massive difference. It makes you wonder if some of these "breakthroughs" are just statistical noise.

Eli: Precisely. They’re proposing we stop looking at just the raw score and start thinking about a "question universe"—this theoretical space of all possible questions—to measure a model's underlying skill.

Nia: I love that mental model. So, let’s dive into how we can use tools like the Central Limit Theorem to finally add some much-needed error bars to these AI evals.

Eli: It is such a cool way to think about it, right? This idea of a "question universe." Instead of just looking at the fifty or a hundred questions in a specific benchmark as the beginning and end of what a model knows, we imagine those questions are just a tiny, tiny sample—like a few drops of water taken from a vast ocean of every possible question you could ever ask a model about that topic.

Nia: That really shifts the perspective. Because if you only care about those specific drops, you are just measuring performance on a test. But if you are trying to measure "intelligence" or "coding ability" or "legal knowledge," you are really trying to figure out what is happening in the whole ocean.

Eli: Exactly. And that is where the statistical heavy lifting comes in. Anthropic’s research, which came out late last year, argues that our real object of interest shouldn't be the "observed average"—the score the model actually got on the test—but the "theoretical average" across that entire question universe. They want to measure the underlying skill, which exists independently of whether the specific questions chosen for the benchmark were particularly easy or hard for that specific model.

Nia: So, it is like the difference between a student who happens to study the three topics that show up on the final exam versus a student who actually understands the entire subject. The first one might get a higher score, but the second one has the actual skill we care about.

Eli: That is a perfect analogy. And to get to that "theoretical average," they suggest we lean on something every stats student learns: the Central Limit Theorem. Now, I know that sounds a bit academic, but the takeaway is actually very practical. It basically says that if you take enough random samples from a distribution—in this case, our question universe—the means of those samples will follow a normal distribution, that classic bell curve.

Nia: Okay, so if we ran a bunch of different versions of the same eval, the scores would cluster around that true, underlying skill level.

Eli: Right. And the "width" of that bell curve tells us exactly how much we can trust the score. That width is known as the Standard Error of the Mean, or SEM. Anthropic is pushing for every researcher to report the SEM alongside every single eval score. Without it, a score of 75% is just a number. With it, we can say, "We are 95% confident the model’s true skill is between 72% and 78%."

Nia: I see. So, if Model A gets a 75% and Model B gets a 77%, but their confidence intervals are wide and overlapping—say, both could be anywhere from 70% to 80%—then we can't actually say Model B is better. It might just be the "luck of the draw" we mentioned.

Eli: Precisely. That 2% gap might be totally meaningless. It’s just noise. By calculating the SEM, we turn that noise into something we can measure. It gives us "analytic robustness." It means if someone else builds a new version of that same test with different but equally difficult questions, we should expect our conclusions about which model is better to actually hold up.

Nia: It feels like we are finally growing up as a field. Moving away from "vibes" and leaderboards that change every week based on tiny margins, and toward actual, rigorous science. But I’m curious—does this work for every kind of test? Because some AI evals feel very different from others.

Eli: That is a great lead-in, because the "standard" way of doing this assumes every question is totally independent—like flipping a coin. But as we know, that’s often not how these benchmarks are actually built.

Nia: Right, because a lot of these tests aren't just a list of random, unrelated trivia questions. I’ve seen plenty of reading comprehension evals where you get a long passage—maybe a short story or a scientific article—and then there are five or six questions all based on that one text.

Eli: Exactly. Evals like DROP, QuAC, or SQuAD work exactly like that. And this is where the "naive" application of the Central Limit Theorem gets dangerous. If you treat those six questions as six independent data points, you are tricking yourself into thinking you have more information than you actually do.

Nia: Because if the model fails to understand one key metaphor in that passage, it’s probably going to miss all six questions, right? They aren't independent trials; they are all tied to the same "unit" of the passage.

Eli: You've hit the nail on the head. Anthropic points out that if the questions are related in clusters, the "luck of the draw" becomes even more exaggerated. If you happen to pick a passage about 18th-century maritime law and the model happens to be weak on that specific niche, your score drops significantly across multiple questions at once.

Nia: So, the variance is much higher than it looks on paper.

Eli: Much higher. In their paper, they found that when you "cluster" the standard errors—meaning you treat the passage, not the individual question, as the primary unit of randomization—the error bars can be over three times as large as the naive version. Imagine thinking your margin of error is 1% when it’s actually 3.5%. You’d be declaring winners in "model wars" that are actually just ties.

Nia: That is a huge wake-up call for anyone obsessing over those leaderboard rankings. If you aren't clustering your standard errors on these reading tasks, you are essentially seeing patterns in the clouds.

Eli: It really is. And this isn't some brand-new math Anthropic invented; they are actually pulling these techniques from the social sciences, where researchers have dealt with "clustered" data for decades—like studying students within classrooms or patients within hospitals. They are basically saying, "Hey, AI research community, let’s use the tools that already exist to make our measurements more honest."

Nia: It’s interesting how "honest" is the right word there. It’s not about making the model look better or worse; it’s about acknowledging the limits of what we actually know from the data we have.

Eli: Exactly. And once you acknowledge that noise, the next logical step is to figure out how to reduce it. If our measurements are too "spread out" or "noisy," how do we tighten that bell curve? How do we make the measurement more precise without just having to write ten thousand new questions?

Nia: I’m guessing that’s where we get into the "within-question" variance? Because models don't always give the same answer to the same question twice, do they?

Eli: They don't! Unless you turn the temperature down to zero, there is always a bit of randomness. And Anthropic has some really clever suggestions for how to deal with that, depending on whether the model is "thinking" out loud or just giving a straight answer.

Nia: So, let’s talk about that randomness. If I ask a model the same math problem five times, it might give me the right answer four times and get lost in the weeds once. That "random component" is another source of noise that can mess up our eval scores.

Eli: It really is. Anthropic breaks the score for any single question into two parts. First, there is the "mean score"—which is the average you’d get if you asked that same question an infinite number of times. That represents the model's "true" ability on that specific question. Then, there is the "random component"—the difference between a single answer and that true average.

Nia: It’s like if I’m a 70% free-throw shooter. My "mean score" is 0.7. But on any single shot, I either make it or I miss. The "random component" is the gap between my actual shot—1 or 0—and my 0.7 average.

Eli: That is a perfect way to put it. And because of the "law of total variance," if we can shrink that random component, we automatically shrink the standard error of the whole eval. We get a clearer picture of the model's actual skill.

Nia: So how do we do that? Do we just ask the model the same question over and over again?

Eli: Well, it depends on the "path" the model takes. If the eval uses "Chain of Thought" reasoning—where the model writes out its logic before giving a final answer—Anthropic recommends exactly that: resampling. You ask the question multiple times, maybe five or ten, and then you take the average of those attempts as the score for that question.

Nia: That makes sense. It smooths out those "oops" moments where the model just took a wrong turn in its reasoning. I noticed they mentioned that the "Inspect" framework—which is an open-source tool for evals—already handles this through something called an "epochs" parameter. It’s nice to see these theoretical recommendations already being baked into the tools people use.

Eli: It really helps with the "path dependency" problem. But what if the model isn't doing Chain of Thought? What if it’s just a multiple-choice question where the model just outputs "A", "B", or "C"?

Nia: Do you still need to resample then? That seems like it would be a lot of extra compute for the same result.

Eli: You actually don't! And this is one of my favorite "pro tips" from the paper. If the model is giving a direct answer, you can often eliminate the random component entirely by looking at the "next-token probabilities."

Nia: Wait, explain that. How does looking at the probability replace the need for multiple runs?

Eli: Think about it this way: instead of making the model pick one answer and seeing if it’s right, you look under the hood at how "confident" the model was in each choice. If the correct answer is "B," and the model’s internal probability for "B" was 0.85, you just record 0.85 as the score for that question. You don't need to actually "roll the dice" and let the model generate a token. You are directly measuring its internal state.

Nia: Oh, that’s brilliant! It’s like instead of asking the 70% free-throw shooter to actually shoot, you just look at their "skill stat" and write down 0.7. You get the perfect average instantly without any of the "luck" of the actual shot.

Eli: Exactly! It’s way more efficient and it totally removes that layer of noise. Interestingly, Anthropic noted that they aren't aware of any major open-source eval frameworks that actually implement this technique yet. It’s a huge opportunity for the community to get much more precise data with less work.

Nia: It’s funny how we often treat these models like humans—we want to see them "take the test"—when they are actually these probabilistic engines that we can probe directly. Using the probabilities feels like using the model’s "true" voice.

Eli: It really does. But even if we do all this—if we cluster our errors and we use probabilities to reduce noise—we still have the problem of comparing two models. And that’s where things get really interesting, because models tend to have some "hidden" similarities that we can use to our advantage.

Nia: I’ve noticed that when a new model comes out, people immediately start comparing it to the previous "king of the hill." It’s always "Model X vs Model Y." And usually, they just put the two scores side-by-side.

Eli: Right, but that side-by-side comparison actually throws away a lot of really valuable information. Think about it: Model A and Model B are taking the exact same test. They are facing the exact same set of questions.

Nia: So, if a question is incredibly difficult—like a really obscure math problem—both models are likely to struggle with it.

Eli: Exactly. In their paper, Anthropic found that "frontier" models—the really top-tier ones—have a substantial correlation in their scores. Usually between 0.3 and 0.7 on a scale of -1 to 1. That means they tend to get the same questions right and the same questions wrong. They have similar "blind spots."

Nia: That makes total sense. They are often trained on similar datasets, so they might struggle with the same weird edge cases or poorly phrased questions in the benchmark.

Eli: Precisely. And if we ignore that, we are keeping all the "noise" of the question difficulty in our comparison. But if we use a "paired-differences test," we can actually cancel out that noise. Instead of looking at "Model A's score" and "Model B's score," we look at the difference between their scores on every single question.

Nia: Oh, I see! So if a question is just bad or confusing, and both models get a low score, the difference is zero. The "noise" of that bad question just vanishes from the comparison.

Eli: You've got it! It lets us focus purely on the variance in their responses rather than the variance in the questions. Anthropic calls this a "free" variance reduction technique. It doesn't cost any extra compute, it doesn't require more questions—you just have to change the way you analyze the data you already have.

Nia: It’s like comparing two runners. If you just look at their times on different days, you have to worry about the wind, the temperature, the track quality. But if you have them run side-by-side on the same track at the same time, all those external factors cancel out. You are just measuring the difference in their speed.

Eli: That is a perfect analogy. And when the correlation between the models is high—which it is for these top-tier models—the standard error of that "mean difference" becomes much smaller. It makes the "signal" of which model is better much, much clearer.

Nia: So, Anthropic is recommending that we always report these pairwise stats—the mean difference, the confidence interval of that difference, and that Pearson correlation coefficient. It’s a much more sophisticated way of saying "Model A is better than Model B."

Eli: It really moves us away from "Model A got 80 and Model B got 81" toward "Model A outperforms Model B by 1% with a 95% confidence interval of 0.5% to 1.5%." That second statement is actually meaningful science.

Nia: It also feels like it would stop a lot of the "hype cycles" where a model is claimed to be "the new leader" based on a tiny, statistically insignificant margin. If the confidence interval for the difference includes zero, then they are essentially tied, regardless of what the raw numbers say.

Eli: Exactly. It’s about having the discipline to say "we don't know yet" when the data is too noisy. But how do we know if we have enough data in the first place? That’s where we get into the "flip side" of significance, which is statistical power.

Nia: "Statistical power." That’s one of those terms that sounds very authoritative, but I think a lot of people—myself included—might be a little fuzzy on what it actually means in practice.

Eli: It’s actually pretty intuitive once you break it down. If statistical significance is about avoiding "false positives"—thinking there’s a difference when there isn't—then statistical power is about avoiding "false negatives." It’s the ability of your test to actually detect a difference, assuming one really exists.

Nia: Ah, okay. So if I’m testing a new medicine that works, but I only test it on three people, I might not see a "statistically significant" result just because my sample size is too small. I didn't have enough "power" to find the truth.

Eli: Exactly! And the same thing happens with AI evals. If an eval only has fifty questions, the confidence intervals are going to be huge. You’d need one model to be way better than the other—like 15 or 20 points better—for the test to actually register a "statistically significant" win.

Nia: So, if Model A is actually 3% better than Model B, but I only use a small benchmark, I might walk away thinking they are tied. I missed the "signal" because my "microscope" wasn't powerful enough.

Eli: That is exactly right. Anthropic’s paper shows researchers how to use "power analysis" to plan their evals before they even run them. It’s a mathematical relationship between four things: the number of questions (the sample size), the "power" (the probability of finding the difference), the "false positive rate" (usually set at 5%), and the "effect size"—which is how much of a difference you are trying to detect.

Nia: So, if I’m a researcher and I think my new model is about 2% better at coding, I can use this formula to figure out exactly how many coding questions I need in my benchmark to actually prove it?

Eli: Precisely. It keeps you from wasting time on evals that are "underpowered." If the formula says you need two thousand questions to detect a 2% difference, and you only have five hundred, you know right away that your results are going to be inconclusive. You either need more questions, or you need to use those variance-reduction techniques we talked about—like resampling or looking at probabilities—to "boost" your power.

Nia: I love how this turns "evaluating AI" into a much more deliberate engineering task. You aren't just throwing questions at a model and hoping for the best; you are designing an experiment with a specific goal and a specific level of precision in mind.

Eli: It also helps with the "subsampling" problem. Sometimes an eval is huge—like thousands of questions—and it’s too expensive to run the whole thing every time you make a change to the model. Power analysis tells you how many questions you can "randomly subsample" while still being able to see if your changes are actually making the model better.

Nia: It’s all about efficiency and clarity. It feels like Anthropic is giving the community a "flight manual" for how to navigate the messy world of model performance. But I’m wondering, for the average person who isn't a researcher—maybe a developer trying to choose between two models—how does this change how they should look at all these charts and tables they see online?

Eli: It really changes the "filter" through which we should view everything. It’s about looking for those error bars and asking, "Is this difference real, or is it just the luck of the draw?"

Nia: So, if I’m a developer or even just an AI enthusiast, and I’m looking at a brand-new model launch—let's say it’s March 2026 and everyone is buzzing about the latest "Model Z"—what are the "red flags" or "green flags" I should be looking for in their eval reports?

Eli: That’s a great way to frame it. The first "green flag" is definitely the presence of error bars. If a company reports a single percentage point without a Standard Error of the Mean or a 95% confidence interval, you should immediately be skeptical. They are giving you the "observed average" but hiding the uncertainty.

Nia: Right. It’s like a political poll that says "Candidate A is at 48%" but doesn't tell you the margin of error is 5%. Without that margin, the 48% is basically a guess.

Eli: Exactly. The second thing to look for is "clustering." If the benchmark involves reading long passages or solving multi-step problems that are grouped together, did they account for that? If they didn't cluster their standard errors, their "confidence" is likely way overblown. They might be reporting a "significant" lead that would vanish if they just did the math correctly.

Nia: And what about those "paired comparisons"? If they are comparing their new model to a competitor, are they showing us the "mean difference" and the correlation?

Eli: Yes! That is a huge one. If they just show two separate bars on a graph, they aren't taking advantage of the "free" variance reduction. A "green flag" report would show a distribution of the difference between the two models. That tells you not just that it’s better, but how consistently it’s better across different types of questions.

Nia: I also think the "Chain of Thought" point is really practical for developers. If you are building an app and trying to decide if a new prompt version is better, you should be doing that resampling Anthropic mentioned. Don't just run your test suite once. Run it a few times and average the scores for each "test case."

Eli: Absolutely. And if you have access to the log probabilities—the "logprobs"—of the model’s output, use them! It’s the most precise signal you can get. Instead of a "pass/fail" on a test, you get a "probability of passing," which is much more stable and requires fewer runs to get a clear answer.

Nia: It really comes down to acknowledging that these models are "stochastic"—they have a bit of a "roll the dice" quality to them. Our evaluation methods need to respect that randomness rather than trying to ignore it.

Eli: It’s the difference between "vibe-based" engineering and actual empirical science. Anthropic’s paper is a call to action for the whole industry to stop being so sloppy with our measurements. Because as they put it, an "empirical science is only as good as its measuring tools."

Nia: "Only as good as its measuring tools." I love that. It reminds me that as much as we talk about the "magic" of AI, at the end of the day, it’s a technology that we have to be able to measure and understand if we want to build things that actually work in the real world.

Eli: Well said. And while statistics might not be the entire science of AI evaluation, it’s a critical foundation. It’s the language we use to talk about noise, uncertainty, and truth.

Nia: You know, Eli, looking at how much the field has changed even in the last year—it’s March 2026 now, and the models we are seeing are just light-years beyond what we had when this Anthropic paper first came out in late 2024. But the math hasn't changed. These statistical principles are just as relevant today as they were then.

Eli: It’s true. The models get bigger, the "question universe" gets more complex, but the need for rigorous measurement only grows. In fact, as the models get closer and closer in performance—as we reach these highly competitive "frontiers"—those tiny 1% or 2% differences matter more than ever.

Nia: It’s like Olympic sprinting. When everyone is running a 10-second hundred-meter dash, you need a very precise clock to figure out who actually won. You can't just use a handheld stopwatch and expect to see the difference.

Eli: That’s a perfect way to put it. We are in the "Olympic era" of AI development, and our "stopwatches"—our benchmarks and eval frameworks—need to be incredibly precise. Anthropic’s recommendations on clustering, paired differences, and power analysis are essentially the "high-speed cameras" of the AI world.

Nia: It also makes me think about the "Science Blog" Anthropic launched around the same time. They are really pushing this idea that AI research should look more like traditional science—more transparency, more focus on methodology, more collaboration with external labs. It’s not just about "launching a product"; it’s about "understanding a phenomenon."

Eli: I think that’s a really healthy shift. We’ve had a few years of "move fast and break things," but now we are seeing the value of "move carefully and measure things." Whether it’s their work on "long-running Claude for scientific computing" or these statistical deep-dives, there’s a clear focus on making AI a tool that scientists can actually trust for multi-day, complex tasks.

Nia: And you can't have trust without reliability. You can't have reliability without knowing your margin of error. It all comes back to those "error bars" we talked about at the beginning.

Eli: It really does. It’s the difference between a "cool demo" and a "reliable system." For everyone listening who is building with these models, that is the takeaway: don't be afraid of the uncertainty. Embrace it, measure it, and use it to build something better.

Nia: I think that’s a perfect place to leave it. We’ve gone from the "luck of the draw" to the "question universe," and from "naive averages" to "clustered standard errors." It’s been a bit of a journey through the stats textbook, but it’s so clearly worth it.

Eli: It really is. And honestly, it makes the whole field feel a lot more grounded. It’s not just "magic boxes"; it’s math, it’s data, and it’s something we can actually get a handle on.

Nia: So, as we wrap things up today, I’m left thinking about that phrase "the science of measurement in the presence of noise." It’s such a beautiful definition of statistics. In a world where AI can feel like it’s changing every single day, having those "measuring tools" feels like having a compass in a storm.

Eli: I love that. And for our listeners, I’d encourage you to think about the next AI "breakthrough" you see on social media or in the news. Don't just look at the headline score. Ask yourself: "Where are the error bars? Was this a paired comparison? Did they account for the luck of the draw?"

Nia: It’s a great way to become a more critical consumer of AI news. And if you are a developer or a researcher yourself, maybe take one of these ideas—like using "next-token probabilities" for your evals or running a quick power analysis—and try applying it to your next project. It might just change how you see your model's performance.

Eli: It definitely will. It’s all about finding the signal in the noise. And once you start looking for it, you realize the noise is everywhere—but so are the tools to handle it.

Nia: Exactly. Thank you all for joining us for this deep dive into the "science of evals." It’s been a fascinating look under the hood of how we actually measure the "intelligence" we are all so excited about.

Eli: It really has. Thanks for the great questions, Nia. It’s been a blast.

Nia: To everyone listening, take a moment to reflect on how you measure "success" in your own work with AI. Is it just a "vibe," or could a little bit of statistics help you see things more clearly? We hope this conversation gives you a few more tools for your toolbox. Thanks for listening.