Why AI benchmarks are more uncertain than they look

Nia: Jackson, I was looking at some AI benchmarks earlier, and it hit me—we always see these bolded "state-of-the-art" scores, but we almost never see error bars. It’s like we’re just assuming the model didn’t just get lucky with a specific set of questions!

Jackson: Right! And that’s exactly what Anthropic’s recent research tackles. They found that on some popular evals, if you don't account for how questions are grouped together, your standard error could be off by over three times.

Nia: Wait, three times? That’s a massive gap. It means a model that looks like a winner might actually just be tied with its predecessor once you look at the "question universe" it's drawing from.

Jackson: Exactly. It’s about measuring underlying skill, not just the luck of the draw. They’re proposing a whole new way to report these results using the Central Limit Theorem to bring some much-needed rigor to the field.

Nia: I love that. So, let’s dive into how these statistical "error bars" actually change the way we rank these models.

Jackson: Before we get into the nitty-gritty of the math, we really have to wrap our heads around this concept of the "question universe." It sounds like something out of a sci-fi novel—doesn't it—but it’s actually the cornerstone of their whole statistical approach.

Nia: It definitely has a cinematic ring to it! But in practical terms—what are we actually talking about when we say "question universe"? Are we saying there’s an infinite list of math problems floating around in digital space?

Jackson: In a sense—yes. When a researcher builds an evaluation like MMLU or MATH—they aren't capturing every possible question that could ever exist on those topics. They’re picking a sample. Maybe it’s a thousand questions—maybe it’s five thousand. But the "true" skill we’re trying to measure isn't just "how well can this model answer these specific five thousand questions." It’s "how well does this model understand the underlying principles of physics—or law—or coding."

Nia: Right—because if I just memorize the answers to one specific test—I haven't actually learned the subject. I’ve just learned that test. So the "question universe" is the total—theoretical sum of all possible questions that represent a skill.

Jackson: Exactly. And the big shift Anthropic is pushing for is that we should stop treating the observed average—the score you see on the leaderboard—as the final truth. Instead—we should view it as an estimate of the model’s performance across that entire—unseen super-population of questions.

Nia: That makes so much sense. It’s like if I’m a scout looking for a basketball player. I don't just care if they made ten shots in a row today—I want to know their "true" shooting percentage—their underlying ability—which would reveal itself if they took ten thousand shots.

Jackson: That’s a perfect analogy. And this is where the Central Limit Theorem—or CLT—comes into play. The researchers point out that because these evaluations are usually made up of hundreds or thousands of unrelated—independent questions—the scores actually follow a normal distribution. You know—the classic bell curve.

Nia: So—if we imagine drawing multiple different samples of questions from that "universe"—the averages of those samples would cluster around the "true" mean?

Jackson: Precisely. And the width of that bell curve tells us exactly how much "luck of the draw" is involved. That width is what statisticians call the Standard Error of the Mean—or SEM. Anthropic’s first big recommendation is that every single eval score should be reported with its SEM. Without it—we have no idea if a 75 percent score is a solid—reliable measurement or just a lucky spike.

Nia: It feels like we’ve been flying blind without this. If one model gets a 65 percent and another gets a 67 percent—everyone screams "New SOTA!" But if the standard error is—say—3 percent—then that 2 percent lead is essentially meaningless. It’s just noise.

Jackson: You’ve hit the nail on the head. In their paper—they show how you can use that SEM to create a 95 percent confidence interval. You just take the mean—and add or subtract 1.96 times the standard error. If your "winning" model’s interval overlaps significantly with the "losing" model’s interval—you can’t actually claim victory.

Nia: It’s a reality check for the industry. It’s moving us away from this "highest number wins" mentality and toward actual scientific rigor. But wait—you mentioned earlier that sometimes questions aren't independent. How does that mess with the bell curve?

Jackson: That brings us to one of the most fascinating—and honestly—most overlooked parts of their research. It’s the problem of "clustering." And if you ignore it—your error bars are going to be way too small—giving you a false sense of confidence.

Nia: Okay—let's talk about this clustering thing. I’m thinking about a reading comprehension test. Usually—you get a long passage about—I don't know—the history of the steam engine—and then you have five or six questions all based on that one passage. That’s what we mean by a cluster—right?

Jackson: Exactly. Evals like DROP—QuAC—RACE—and SQuAD all work this way. And here’s why that matters for the math: if a model finds the passage about steam engines really confusing—it’s probably going to miss all five questions. If it finds the passage easy—it’ll probably get all five right.

Nia: So the questions aren't "independent draws" from the universe anymore. They’re tied together by the passage. It’s like if I ask you five questions about your own childhood—if you remember that time period—you’ll get them all—but if you’ve blocked it out—you’ll miss all of them. One "draw" of a topic determines multiple outcomes.

Jackson: Right. And the Central Limit Theorem—in its simplest form—assumes every question is a totally independent event. If you apply that "naive" math to a clustered eval—you’re basically double-counting—or triple-counting—the information you’re getting. You think you have a thousand independent data points—but you really only have—say—two hundred independent passages.

Nia: Oh—I see. So your "N"—the number of things you’re actually testing—is much smaller than it looks on paper.

Jackson: Exactly. And when "N" is smaller—your uncertainty is larger. Anthropic’s researchers found that on popular reading comprehension evals—the "real" standard error—the one that accounts for these clusters—can be more than three times larger than the naive version.

Nia: Three times! That is a huge correction. If I’m a researcher and I think my error bar is 1 percent—but it’s actually 3 percent—I might be publishing results that are basically just statistical phantoms.

Jackson: It happens more than you’d think. They actually pointed to the Llama 3 technical report as an example. They noted that the confidence intervals reported there were likely too narrow in some cases because they didn't account for this clustering—and too wide in others because they used a simpler Bernoulli formula that doesn't handle fractional scores—like F1 scores—very well.

Nia: So—even the biggest players in the game are still refining how they do this. It sounds like Anthropic is trying to borrow a "playbook" from the social sciences—where they’ve been dealing with clustered data in things like school district studies for decades.

Jackson: That’s exactly what they’re doing. They’re recommending that we "cluster" the standard errors on the unit of randomization. So—if the passage is what’s being randomly selected—that’s your cluster. If you’re doing a multilingual eval like MGSM—where you translate the same math problem into ten different languages—the original problem is the cluster.

Nia: Because if the model can't solve the core math logic—it doesn't matter what language you ask it in. It’s the same "knowledge gap" being tested ten times.

Jackson: Precisely. They even provided a specific formula—which we can dig into—that acts as a "sliding scale." It looks at how correlated the answers are within a cluster. If they’re perfectly correlated—the cluster acts as one single observation. If they’re totally unrelated—it behaves like the standard independent math.

Nia: It feels like this is about honesty in reporting. It’s saying—"Hey—don't tell me you’re 99 percent sure about this model’s superiority if your test questions are all bunched up around a few topics."

Jackson: Right—and it goes beyond just being "honest." It’s about being efficient. Once you realize how much noise is in the system—you start looking for ways to turn the volume down. And that leads us to some really clever "variance reduction" techniques that don't involve just writing ten thousand more questions.

Nia: So—if the goal is to make our measurements more precise—to shrink those error bars—the obvious answer is just "more questions"—right? But I imagine writing high-quality—vetted AI eval questions is incredibly expensive and slow.

Jackson: It’s a massive bottleneck. That’s why Anthropic’s third recommendation is so critical. They’re looking for ways to get more "signal" out of the questions we already have. They call this reducing the "conditional variance."

Nia: "Conditional variance"—okay—break that down for me.

Jackson: Think of a model’s score on a single question as having two parts. Part one is the "true" mean—the average score the model would get if you asked it that exact same question an infinite number of times. Part two is the "random component"—the "luck" of that specific run.

Nia: Wait—why would a model give different answers to the exact same question? Aren't they deterministic?

Jackson: Well—usually we run them with some "temperature"—which introduces randomness in the token selection to make the output more creative or varied. So—on a tough coding question—the model might "hallucinate" a bug one time and write perfect code the next—even with the same prompt.

Nia: Ah—so that’s the noise. One single "pass" or "fail" on a question might not represent the model’s true capability on that specific problem. It might have just rolled a one on its virtual dice.

Jackson: Exactly. So—the researchers suggest two main ways to kill that noise. The first is "resampling." Instead of asking the question once—ask it—say—ten times. Then—take the average of those ten attempts as the score for that question.

Nia: Oh—I see! That "question-level average" is going to be a much more stable—reliable number than a single 1 or 0. It’s like letting a golfer take five putts and averaging the results instead of just judging them on one swing.

Jackson: Right. And they actually worked out the math on this. They showed that if you move from one sample to—say—four samples per question—you can reduce the total variance of your eval by half in some cases. There’s a "point of diminishing returns"—though. Once your resampling noise is much smaller than the "question difficulty" variance—adding more samples doesn't really help you much.

Nia: That’s a great practical tip for anyone running these. But what about models that don't use "Chain of Thought"? I know a lot of evals are just simple multiple-choice questions.

Jackson: That’s where the second technique comes in—and it’s incredibly elegant. They suggest using "next-token probabilities." Instead of making the model actually "pick" an answer—like "A" or "B"—and then seeing if it got it right—you just look at the internal probability the model assigned to the correct token.

Nia: Wait—so if the correct answer is "B"—and the model’s internal math says there’s a 72 percent chance the next token should be "B"—we just give it a score of 0.72?

Jackson: Exactly! You’ve completely eliminated the "randomness" of the actual generation. You’re looking directly at the model’s "certainty." It’s the ultimate variance reduction because the "random component" becomes zero.

Nia: That’s brilliant. It’s like being able to read the student’s mind during a test to see how well they knew the answer—rather than just looking at the bubble they filled in. You get a much more nuanced—continuous score.

Jackson: It really is. And they point out that doing this can reduce the variance by two-thirds compared to just grading a single sample. But—there’s a big "don't" here. A lot of people think—"Hey—if I want to reduce randomness—I’ll just turn the temperature down to zero—make the model deterministic."

Nia: That seems like the easy fix. Why is that a bad idea?

Jackson: Because it can actually backfire and triple your variance! When you force the model to "round" its internal probabilities to the single most likely answer—you’re losing all that nuanced information. You’re essentially turning a smooth distribution into a bunch of jagged blocks.

Nia: Plus—it might change how the model actually performs—right? It’s not the same "version" of the model you’d use in the real world.

Jackson: Exactly. You’re injecting bias. The goal isn't to change the model to fit the math—it’s to use better math to measure the model as it actually is. And speaking of "better math"—the way we compare two different models is another area where a simple tweak can give us a massive boost in precision. It’s all about "paired differences."

Nia: Okay—so we’ve talked about making a single model’s score more accurate. But—let’s be real—the thing everyone actually cares about is: "Is Model A better than Model B?"

Jackson: That’s the "billion-dollar question"—literally. And the standard way people do this is "unpaired analysis." They take Model A’s score—Model B’s score—and they look at the gap. But Anthropic argues this is a huge missed opportunity.

Nia: Why? If I have both scores—what am I missing?

Jackson: You’re missing the "hidden structure" of the data. Because—usually—both models are being tested on the exact same list of questions.

Nia: Right—that’s the whole point of a benchmark. It’s the same "test paper."

Jackson: Exactly. And because it’s the same test paper—we can look at the "paired difference" for every single question. We can ask: "On question number 42—did Model A beat Model B?"

Nia: Oh—I see where this is going! This lets you ignore how "hard" the questions are and just focus on the gap between the models.

Jackson: You’ve got it. Think about it this way—if an eval has some really—really hard questions that both models miss—and some really easy questions that both models get right—those questions are "noisy." They don't help us distinguish between the two models—but they still add to the overall variance of the individual scores.

Nia: Right—they’re just "padding" that makes the error bars look wider than they need to be.

Jackson: But—when you do a paired test—you’re essentially "subtracting out" the difficulty of the questions. You’re looking at the correlation between the models. Anthropic found that "frontier" models—like the ones they develop or the ones from OpenAI—actually have a high correlation. They tend to get the same questions right and wrong.

Nia: That makes sense. They’re trained on similar data—they have similar strengths.

Jackson: Exactly. They found correlation coefficients between 0.3 and 0.7. And when models are correlated like that—using a paired-difference test gives you a "free" reduction in your standard error. It makes your measurement of the difference much—much more precise.

Nia: It’s like if two people are racing—you don't just look at their finish times on different days. You have them run side-by-side on the same track—in the same wind conditions. The "track conditions" cancel out—and you just see who’s faster.

Jackson: That’s a perfect way to put it. They even provided a fictional example in the paper—comparing two models they called "Galleon" and "Dreadnought." On the surface—Dreadnought won two out of three evals. It looked like the clear winner.

Nia: I remember that from the intro—it seemed like Dreadnought was the one to go with.

Jackson: But—once they ran the paired-difference analysis—the truth came out. It turned out that Galleon’s victory on the first eval—MATH—was actually statistically significant. But Dreadnought’s "victories" on the other two? They were within the margin of error. They were just noise.

Nia: So—the "winner" was actually the model that lost two out of three benchmarks? That is a total 180!

Jackson: It really is. It shows how dangerous it is to just look at raw percentages. If you don't have those "paired" error bars—you’re basically just flipping coins and calling it science.

Nia: This feels like it should be the new gold standard for every technical report. Don't just give us a table of scores—give us the pairwise differences—the correlations—and the paired standard errors.

Jackson: Absolutely. And that leads to the final piece of the puzzle—which is "Power Analysis." It’s how you figure out—before you even start—if your eval is actually capable of telling the difference between two models. It’s about planning the experiment so you don't waste your time.

Nia: "Power Analysis." I’ve heard that term in medical studies—like when they’re trying to figure out how many people need to be in a drug trial. How does that translate to AI?

Jackson: It’s the exact same logic. "Power" is basically the probability that your test will actually detect a difference—assuming one really exists. If your eval only has ten questions—it doesn't matter how much better Model A is—the "noise" of those ten questions is going to be so high that you’ll probably never get a statistically significant result.

Nia: So—it’s like trying to weigh a diamond on a bathroom scale. The tool just isn't sensitive enough to "see" the signal.

Jackson: Exactly. And Anthropic’s paper gives researchers a "sample-size formula" to fix this. It lets you work backward. You can say: "I want to be able to detect a 3 percent difference in performance—with 80 percent certainty—and a 5 percent false-positive rate. How many questions do I need?"

Nia: That is incredibly useful. Instead of just "hoping" your eval works—you can mathematically prove that it’s big enough to be useful.

Jackson: Right. And they ran some "reasonable" numbers through that formula. They found that for a typical AI eval—you really need at least a thousand independent questions to have good "signaling ability."

Nia: A thousand? Wow. I’ve definitely seen evals with way fewer than that.

Jackson: Me too. And the formula also helps you decide on your "resampling" strategy. Remember how we talked about asking the same question multiple times? The power formula can tell you: "Hey—if you have a small number of questions—you can make up for it by resampling each one ten times." It gives you a way to balance the cost of "more questions" versus "more compute."

Nia: It’s like a budget for your experiment. You can spend your "effort" on more questions or more "samples per question."

Jackson: Precisely. They even mentioned that if an eval is too small—the power analysis might tell you it’s not even worth running. It saves you from drawing false conclusions from a "weak" test.

Nia: I think this is so important because—as these models get better—the gaps between them are getting smaller—right? We’re not looking for 50 percent improvements anymore—we’re looking for 2 or 3 percent.

Jackson: You’re 100 percent right. As we reach the "frontier"—the signal gets quieter and the noise stays the same. That’s why the math has to get better. If we’re chasing 2 percent gains—but our "error bars" are 5 percent wide—we’re just wandering in the dark.

Nia: Okay—Jackson—we’ve covered a lot of ground. If I’m a researcher—or an engineer—or even just an enthusiast trying to make sense of the latest model release—what is the actual "playbook" here? How do we put this into practice?

Jackson: Well—the paper boils it down to a few very clear "marching orders." First—and most importantly—stop reporting raw averages in a vacuum. Every score needs a Standard Error of the Mean—calculated using the Central Limit Theorem.

Nia: And put it in parentheses right under the score! Make it as common as the score itself.

Jackson: Exactly. Second—look for clusters. If your eval has related questions—don't use the "naive" math. Use the clustered standard error formula. If you don't—you’re basically lying to yourself about how precise your data is.

Nia: Third—reduce that variance! If you can use next-token probabilities—do it. It’s the "cleanest" signal you can get. If you can't—because the model needs to "think" with Chain of Thought—then resample. Ask each question multiple times and average the results.

Jackson: And fourth—when you’re comparing models—always—always use paired-difference analysis. It’s a "free" way to shrink your error bars just by using the information you already have—the fact that both models took the same test.

Nia: I also love the "Power Analysis" piece as a pre-flight check. Before you even fire up the GPUs—run the numbers. Is this eval actually capable of answering the question you’re asking? If the "Minimum Detectable Effect" is 10 percent—but you’re looking for a 2 percent improvement—you’re just wasting electricity.

Jackson: It’s about moving from a "contest" mindset to an "experiment" mindset. Instead of "who has the biggest number"—it’s "what can we reliably prove about these capabilities."

Nia: It really changes the way I’m going to look at these leaderboards from now on. I’m going to be squinting—looking for those parentheses and error bars.

Jackson: As you should! And honestly—the more the community adopts these standards—the faster we’ll actually make progress. We’ll stop chasing "phantom" improvements and focus on the gains that are actually real.

Nia: It feels like the "growing up" phase of AI research. We’re moving from the "wild west" of just throwing things at the wall—to actual—rigorous science.

Jackson: That’s a great way to put it. And there’s one final thing the researchers emphasized—which I think is a great "meta-point." They said—"Statistics is the science of measurement in the presence of noise." AI evaluations are—by their nature—incredibly noisy. This isn't about making the noise go away—it’s about learning how to work with it—honestly and precisely.

Nia: "Measurement in the presence of noise"—I love that. It’s so much more realistic than pretending we have "perfect" tests.

Nia: You know—Jackson—this whole conversation has really shifted my perspective. I used to think of AI evals as these definitive—objective grades. Like—"Model X got a B+." But now—I see them more like a political poll or a scientific study. There’s always a margin of error—and that margin is just as important as the result itself.

Jackson: It really is. We have to respect the "noise." If we don't—we’re just creating a false narrative of progress. Anthropic’s work is a call to action for the whole field to treat these evals as "informative experiments" rather than just a series of contests.

Nia: It reminds me of how medicine evolved. They didn't just start with double-blind—placebo-controlled trials. They had to develop the statistical tools to realize that "Hey—maybe this patient got better because they were already healthy—not because of the medicine." We’re doing the same thing for AI "brains."

Jackson: Exactly. We’re moving past the "anecdotal" phase. And what’s exciting is that these tools are available right now. They even mentioned that frameworks like "Inspect" are already starting to build some of this in—calculating standard errors correctly using "epochs."

Nia: So—the "infrastructure" for a more scientific approach is already being built. That’s really encouraging.

Jackson: It is. But it’s going to take a cultural shift too. It’s about being okay with saying—"We don't know for sure who won this benchmark yet—the results are inconclusive." That’s a hard thing to say in such a competitive—fast-moving industry.

Nia: But it’s the only way to build something that actually lasts—right? If we build our future on statistical noise—the whole thing eventually falls apart.

Jackson: You’ve hit the nail on the head. Science is only as good as its measuring tools. And with these recommendations—we’re making those tools a lot more sharp.

Nia: Well—Jackson—this has been an absolute masterclass. I feel like I’ve just been given a "pair of glasses" that lets me see through the hype and into the actual data.

Jackson: I’m so glad! It’s a fascinating—evolving field—and honestly—we’re just at the beginning of developing a true "science of evaluations."

Nia: I can't wait to see how this changes the reports we’ll be reading in the coming months. To everyone listening—thanks for joining us on this deep dive into the "math of AI." We hope it gives you a new way to look at the next "SOTA" announcement you see on your feed.

Jackson: Definitely. Take a moment to think about that "question universe" next time you see a score. Is it a solid measurement—or just a lucky draw?

Nia: Such a great question to end on. Thanks for listening—and keep questioning the numbers!