LLM evaluation is noisier than you think

Nia: Eli, I was looking at some recent leaderboard rankings, and it’s wild how we all just hunt for that one model highlighted in bold because it has the highest number. We’ve basically turned LLM evaluation into a high-stakes game of "who has the biggest decimal point."

Eli: It’s the "highest number is best" fallacy, right? But here’s the kicker: according to research from Anthropic, most of those tiny performance gaps we obsess over might just be noise. We’re often mistaking random fluctuations for actual progress because we aren't testing for statistical significance.

Nia: Exactly! It’s like claiming one athlete is better because they ran a millisecond faster once, without checking if they can actually repeat it. It makes you realize that our current approach to evals is actually pretty naive.

Eli: It really is. But the good news is that we can fix this by treating evaluations like the scientific experiments they actually are. We can use tools like the Central Limit Theorem to calculate standard errors and finally put some "error bars" on those scores.

Nia: I love that. It’s time to move past raw metrics and start building a real statistical foundation. Let’s dive into the core tools we need to turn these noisy numbers into reliable insights.

Nia: So, if we’re moving away from just staring at a single accuracy percentage, where do we actually start? I mean, I remember some stats from college—mean, variance, standard deviation—but how do they actually help me decide if my new prompt is better than the old one?

Eli: It’s all about realizing that any evaluation score you see is actually a sample from a "super-population." Think of it this way—there’s an infinite number of ways a user could ask a question, but your evaluation dataset is just a tiny, finite slice of that. When we calculate the mean, or the average score, we’re trying to estimate the "true" performance of the model across all possible questions.

Nia: Right, the sample mean is our best guess, but it’s still just a guess. And I suppose that’s where variance comes in? It tells us how much that guess might wiggle if we ran the test again with different questions?

Eli: Exactly! Variance is the measure of how "spread out" your results are. If your model is super consistent—always getting the easy ones right and the hard ones wrong—the variance might be lower. But if it’s erratic, the variance shoots up. And once you have variance, you can find the standard deviation by taking the square root. That’s your baseline for understanding the "spread" of individual samples.

Nia: Okay, so standard deviation tells me about the individual questions. But when I’m looking at the whole model, I usually care about the average. Is that why researchers talk about "standard error"? I used to get those two mixed up all the time.

Eli: You aren't alone there! It’s a huge point of confusion. Think of it like this: standard deviation is about the diversity of the data points—how much the scores for question A and question B differ. Standard error is about the precision of your average. It’s the standard deviation of the sampling distribution. If you ran your eval a thousand times with different sets of questions, the standard error tells you how much those thousand different averages would vary.

Nia: That makes so much sense. So, a small standard error means I can really trust that my 75% accuracy is actually 75%, and not just a lucky draw of easy questions.

Eli: Spot on. And there’s a beautiful mathematical relationship here. The standard error is basically the standard deviation divided by the square root of the number of samples—your "n." This is the core of the Central Limit Theorem, or CLT. It tells us that as you add more questions to your eval, your sample mean starts to look like a normal distribution—that classic bell curve—centered right on the true performance.

Nia: So "n" is our best friend! If I want to be more certain, I just need more questions. But I noticed in the source materials that for LLMs, we often use Bernoulli variables. That sounds fancy, but it’s just binary "pass or fail," right?

Eli: Exactly. Most evals are "is the answer right or wrong?" That’s a Bernoulli trial. The cool thing is that for binary scores, the variance formula simplifies beautifully to the mean times one minus the mean. It makes calculating that standard error so much faster. You don't even need complex software—you can do it on a napkin if you have the average and the number of questions.

Nia: I love a good napkin calculation. But wait—there was a mention of "covariance" too. How does that fit into the toolkit?

Eli: Covariance is key when we start comparing models. It measures how two variables move together. If Model A and Model B both find the same questions difficult, they have high positive covariance. Recognizing that relationship is actually a "cheat code" for making your comparisons way more precise, which we’ll definitely get into when we talk about paired differences.

Nia: It sounds like we’re building a bridge from "vibes-based" evaluation to actual engineering. But I’m curious—how do we know when "n" is actually big enough for these bell curves to work? Is there a danger in just assuming the math always holds up?

Eli: That is the million-dollar question. The CLT is powerful, but it’s an "asymptotic" property—it kicks in as "n" goes to infinity. In the real world, we’re often working with small datasets, and that’s where the math can start to lie to us if we aren't careful. We have to be aware of the "small data" trap.

Nia: You mentioned that the Central Limit Theorem can "lie" to us if our dataset is too small. That’s a bit terrifying! If I’m running an eval on, say, fifty or a hundred high-quality reasoning problems, am I just flying blind?

Eli: Not totally blind, but you might be wearing blurry glasses. Recent research, specifically a paper titled "Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints," warns that when your "n" is less than a hundred, the CLT-based confidence intervals tend to be too narrow. They make you feel more certain than you actually are.

Nia: So it’s a false sense of security. I see a tight error bar and think, "Great, my model is definitely better," but in reality, the math hasn't "settled" yet because the sample size is too low.

Eli: Exactly. It’s like trying to predict the outcome of an election by asking five people in a coffee shop. Even if you use a fancy formula, the sample size is just too small to reflect the whole country. In the LLM world, benchmarks are getting more specialized. We have things like "SWE-Bench Verified" which has only about five hundred questions. If you start slicing that up into sub-categories, you quickly end up in that danger zone below a hundred samples.

Nia: And if the model gets a perfect score—like 100% or 0%—doesn't that break the math too?

Eli: Oh, absolutely. If every score is a "1," the sample standard deviation becomes zero. If you plug zero into the standard error formula, your error bar disappears! The math tells you there’s zero uncertainty, which is obviously wrong. You just haven't found the questions the model would fail on yet.

Nia: That’s a massive pitfall. So what’s the alternative? If I’m stuck with a small, expensive-to-run dataset, do I just give up on statistics?

Eli: Never! You just switch tools. The experts suggest looking into Bayesian methods. Unlike the CLT, which relies on things "averaging out" over long runs, Bayesian approaches are much more robust for small samples. They allow you to incorporate "priors"—basically a starting assumption about the model's performance—and then update that as you see more data.

Nia: That sounds like a much more "human" way of learning, honestly. But for most of us, if we can't do full Bayesian modeling, the takeaway is just: be skeptical of small datasets. If your "n" is under a hundred, don't bet the company on a 2% improvement.

Eli: Right. And it’s not just about the size of the dataset—it’s about how the questions are related. That leads us to another big blind spot: the independence assumption. Most of these basic formulas assume that every question in your eval is totally independent of the others.

Nia: But they rarely are, right? I mean, if I have ten questions about the same legal document, or five questions translated into different languages, those aren't truly independent.

Eli: Exactly. They’re "clustered." If the model fails to understand that one legal document, it’s probably going to fail all ten questions related to it. If you treat those as ten independent wins or losses, you’re overcounting your evidence. This is why the Anthropic paper emphasizes "clustered standard errors."

Nia: It’s like the effective number of questions is smaller than the actual count. If I have a hundred questions but they’re just ten topics with ten questions each, I might only have the "statistical power" of ten questions.

Eli: You've hit the nail on the head. That’s why reporting the number of clusters—"C"—is just as important as reporting the number of questions—"n." If you ignore the clustering, your confidence intervals will be way too narrow, and you’ll end up mistaking a "cluster-level" fluke for a "model-level" capability.

Nia: This is a great reality check. It’s making me realize that "more data" isn't just a mantra—it’s a requirement for the math to actually work. But what if we can't get more questions? Is there a way to get more "signal" out of the questions we already have?

Eli: There is! And it involves looking under the hood of the model itself—moving from "did it get it right?" to "how sure was it that it was right?"

Nia: Okay, so we've established that more questions are better, but sometimes we’re stuck with what we've got. You mentioned there’s a way to squeeze more "certainty" out of the same number of questions by looking at probabilities. How does that work?

Eli: This is one of the most powerful variance-reduction techniques available. Usually, we look at the "hard" output—the model says "Choice A" or "Choice B." That’s a binary score: 1 or 0. But behind the scenes, the model is actually generating a probability distribution over those tokens. It might be 51% sure it’s "A" or 99% sure it’s "A."

Nia: Oh, I see where this is going! If we just use the 1 or 0, we’re throwing away all that nuance. A "lucky guess" at 51% counts the same as a "confident truth" at 99%.

Eli: Exactly. By using the next-token probabilities directly, we’re essentially calculating the "expected score" for that question. Instead of a noisy 1 or 0, we get a continuous number between zero and one. In the stats world, this removes what we call "within-question variability."

Nia: Wait, "within-question variability"—break that down for me.

Eli: Sure! Think of it like this: if you ask an LLM the same question ten times with a bit of "temperature" or randomness turned on, it might get it right eight times and wrong twice. If you only test it once, you might catch one of those "wrong" times by pure chance. That’s the noise. But if you look at the probability it assigned to the correct answer, you’re seeing the "average" of all those potential runs in one go.

Nia: It’s like seeing the whole distribution instead of just one roll of the dice. The source materials mention that this drastically improves the "Signal-to-Noise Ratio," or SNR.

Eli: It really does. There’s a study that looked at thirteen popular benchmarks and found that switching to these continuous metrics—like using the log-likelihood of the correct answer—made the results much more "monotonic." That’s a fancy way of saying the scores improved more steadily as the models were trained, rather than jumping around randomly.

Nia: That sounds like a dream for an engineer trying to track progress. But what about models where we don't have access to the probabilities? Like those closed-API models?

Eli: That’s the catch. If the "black box" only gives you text, you can't get those raw probabilities. In those cases, the recommendation is "resampling." You ask the same question "K" times—maybe five or ten—and take the average score for that question. It’s not as perfect as having the probabilities, but it still slashes that within-question variance.

Nia: So the rule of thumb is: use probabilities if you can, and if you can't, ask the question multiple times. But I have to ask—doesn't asking the same question multiple times get expensive?

Eli: It can! But here’s the trade-off: would you rather spend more on API calls and actually trust your results, or spend less and make a wrong decision that costs you months of development time? The math shows that increasing "K"—the number of resamples—helps until the within-question noise is much smaller than the actual difference in question difficulty.

Nia: That’s a great way to frame it. It’s an investment in reliability. I also saw a mention of "MMLU-Cloze." Was that an attempt to make a popular benchmark more "continuous"?

Eli: Precisely. MMLU is usually multiple choice—A, B, C, or D. Researchers found that if they turned it into a "cloze" task—where the model has to fill in a blank—and measured the probability of the correct word, the variance dropped significantly. It turned a noisy leaderboard into a much more stable signal.

Nia: It’s interesting how even the format of the question impacts the statistical reliability. It makes me wonder—if we can reduce noise for one model, how do we make sure we aren't being "overly conservative" when we compare two models? Because usually, that’s the real goal—knowing if Model B is better than Model A.

Eli: You've hit on the most important practical application. Comparing models isn't just about looking at two separate error bars and seeing if they overlap. There’s a much more "precise" way to do it using something called "paired differences."

Nia: I want to talk about this "overlap" thing. I’ve definitely been guilty of looking at two models, seeing that their confidence intervals overlap a little bit, and thinking, "Well, I guess we can't say for sure which one is better." But you’re saying that’s actually too conservative?

Eli: It is! This is a super common misconception. If two 95% confidence intervals don't overlap, then yes, the difference is statistically significant. But they can overlap and the difference could still be significant. The better way—the more "precise" way—is to calculate the confidence interval of the difference between the models.

Nia: Okay, walk me through that. Why is the "difference of the means" better than just looking at the means separately?

Eli: Because when you evaluate two models on the same set of questions, their scores are likely correlated. Think about it—an easy question for Model A is probably also an easy question for Model B. A confusing, poorly-worded question will likely trip up both of them.

Nia: Oh, right! So if they’re both "rising and falling" together based on the difficulty of the questions, that "shared noise" cancels out when you look at the difference?

Eli: Exactly! That’s the "cheat code" I mentioned earlier. This is called a "paired difference" analysis. Instead of treating the two models like two independent experiments, you look at each individual question and ask, "How much better did Model B do on this specific prompt compared to Model A?"

Nia: And because they’re facing the same "headwinds" on the same questions, the variance of that difference is often much lower than the variance of the individual models.

Eli: Precisely. The math shows that the variance of the difference is the variance of A plus the variance of B, minus two times their covariance. Since most LLMs have positive covariance—they agree on what’s hard—that "minus" term is a huge win. It shrinks your standard error, which makes your "error bars" on the improvement much tighter.

Nia: So it’s like a "free" reduction in noise just by being smart about how you pair the data. The Anthropic paper actually calls it a "free reduction in estimator variance." I love the idea of getting more certainty without needing more data points.

Eli: It really is a free lunch. In their implementation, they suggest reporting the "score correlation" between the models. If the correlation is high—say, 0.8 or 0.9—it means your paired analysis is going to be incredibly sensitive to even tiny improvements.

Nia: That is so much more useful than a leaderboard. I can imagine a table where instead of just Model A: 70% and Model B: 72%, you see "Mean Difference: 2%, 95% Confidence Interval: 1.5% to 2.5%." That tells me immediately that the 2% isn't just luck.

Eli: Right! If that confidence interval for the difference doesn't include zero, you can be confident Model B is actually an improvement. It takes the guesswork out of the equation.

Nia: But what if I haven't run the eval yet? What if I’m trying to plan an evaluation and I want to know how many questions I need to actually see that 2% difference? Is there a way to do this math in reverse?

Eli: There is, and it’s called "Power Analysis." It’s basically the "pre-game" version of everything we've been talking about. It helps you decide if an experiment is even worth running before you waste a single GPU hour.

Nia: Power analysis sounds like something a superhero would do, but I’m guessing it’s a bit more grounded than that. If I’m planning a new benchmark, how do I use "power" to make sure I’m not just wasting my time?

Eli: Think of "power" as the sensitivity of your test. It’s the probability that you’ll actually detect a difference if one really exists. We’ve all had that frustration where we know a model is better—we can feel it in the responses—but the eval comes back with a "not statistically significant" result.

Nia: Ugh, the dreaded "no signal" result. So power analysis helps us avoid that?

Eli: Exactly. It lets you calculate the required sample size—the "n"—before you start. To do this, you need to decide on three things. First, your "significance level," or alpha—that’s your tolerance for false alarms. Usually, we set that at 5%.

Nia: Okay, so a 5% chance of thinking there’s an improvement when there isn't. What’s the second one?

Eli: The "power" itself, which is one minus beta. Beta is the "missed detection" rate—failing to see a real improvement. A common target is 80% power, meaning you have an 80% chance of catching a real effect.

Nia: And the third?

Eli: The "Minimum Detectable Effect," or delta. This is the smallest improvement you actually care about. If you’re looking for a massive 10% jump, you don't need many questions. But if you’re trying to detect a subtle 1% tweak, you’re going to need a lot of data.

Nia: That makes total sense. If I’m looking for a needle in a haystack, I need a bigger magnet. I saw a formula in the materials that shows the sample size "n" grows quadratically with the inverse of that delta. Does that mean if I want to detect an effect that’s half as small, I need four times the data?

Eli: You nailed it. That’s why detecting tiny improvements is so expensive. If you want to go from detecting a 2% gap to a 1% gap, your "n" doesn't just double—it quadruples. This is a huge "pitfall" for teams that try to run small evals to justify "incremental" progress. The math just isn't on their side.

Nia: So if I only have a hundred questions, I might only be able to reliably detect a 10% difference. If my model only improved by 2%, my eval will probably say "no change" even though the model did get better.

Eli: Exactly! The test just didn't have the "power" to see it. This is why power analysis is so liberating. It tells you, "Hey, don't bother running this 50-question eval for a tiny prompt change. You’re just going to get noise." It saves you from chasing ghosts.

Nia: It also helps with "curating" new datasets, right? If I’m building a new benchmark for, say, medical coding, I can use this formula to figure out exactly how many questions I need to commission to make it a "useful" benchmark for the community.

Eli: Absolutely. And you can also use those variance-reduction techniques we talked about—like resampling or using probabilities—to lower the "n" you need. If you can reduce the noise in each individual question, the "power" of the whole test goes up.

Nia: It’s all connected! The power analysis is the roadmap, and the variance reduction is the fuel. But I’m thinking about the "big picture" now. We've talked about math and formulas, but how does this change the way we actually build models? How does this impact the training process itself?

Eli: It changes everything, because it allows us to distinguish between "signal" and "noise" even during the training run. There’s a whole framework built around this called the SNR framework, which helps us decide which checkpoints to trust.

Nia: We’ve been talking a lot about "post-hoc" evaluation—checking the model after it’s done. But what about during training? I’ve seen those training graphs where the evaluation accuracy just bounces up and down like a heart rate monitor. It’s hard to tell if the model is actually getting better or just having a "good day."

Eli: That "bouncing" is exactly what the "Signal and Noise" framework from the sources tries to solve. When you’re training a model, you’re often doing small-scale experiments—tuning hyperparameters, trying new data mixes—and you need to know if those changes will scale up.

Nia: Right, and if your "small scale" eval is noisy, you might kill a great idea just because it had a bad checkpoint.

Eli: Exactly. The researchers propose a specific way to measure the reliability of a benchmark using a Signal-to-Noise Ratio. They define "signal" as the spread of scores across different models—basically, how well the benchmark can tell the difference between a "good" model and a "bad" one.

Nia: Okay, that makes sense. If every model gets a 50%, the benchmark has zero signal. What about the "noise" part?

Eli: This is clever—they define noise as the variability in performance due to "training randomness." Instead of training ten models from scratch with different seeds—which is way too expensive—they look at the last few checkpoints of a single training run.

Nia: Oh, I see! Since the model is still "vibrating" a bit at the end of training, the standard deviation of those last few checkpoints tells you how much "randomness" is baked into any single score.

Eli: You've got it. So, SNR = Signal / Noise. If a benchmark has a high SNR, it means it’s "sturdy." It can distinguish between models even if there’s a bit of training noise. If the SNR is low, you’re basically just flipping a coin every time you evaluate a checkpoint.

Nia: That is such a practical tool for researchers. They found that benchmarks with high SNR have a much higher "decision accuracy." That basically means if the benchmark says Model A is better, it actually is better in the long run.

Eli: Exactly. And they found some surprising things—like the fact that you can actually subsample a benchmark to make it more reliable. For MMLU, which has fifty-seven tasks, they found that picking just the sixteen tasks with the highest SNR actually made the evaluation more accurate and much cheaper.

Nia: Wait, so "less is more" if the "less" you’re using is higher quality? That’s wild. It’s like trimming the fat off the evaluation.

Eli: It really is. They also suggest a "two-checkpoint" rule—instead of just evaluating the very last checkpoint, average the scores of the last few. It’s a simple form of variance reduction that smooths out those "lucky" or "unlucky" jumps at the end of training.

Nia: It’s like we’re building a multi-layered defense against randomness. We have probabilities to reduce within-question noise, paired differences to cancel out question difficulty noise, and checkpoint averaging to reduce training noise.

Eli: When you stack all those up, you move from "I think this model is better" to "I have a 95% certainty that this model is exactly 2.4% better." That’s the difference between "vibes" and engineering.

Nia: I love that. It makes the whole process feel so much more professional. But I’m realizing that this isn't just about "better" math—it’s about better decisions. If we don't understand the uncertainty, we make bad choices about which models to deploy.

Eli: That’s the core of it. We’re often comparing models across thirteen different benchmarks, and it can be overwhelming. How do we summarize all this data without losing the statistical "truth" in the process?

Nia: We’ve covered so much ground—from the CLT and standard errors to power analysis and SNR. If I’m an engineer listening to this and I want to start applying this "tomorrow," what’s the playbook? How do I actually change my workflow?

Eli: The first step is simple but radical: stop reporting single numbers. Every time you show a mean score, it should be accompanied by a standard error and the "n." It’s about making uncertainty visible. If you’re using a common spreadsheet or a bit of Python code, calculating that standard error—especially for binary scores—is just a one-line change.

Nia: Okay, "no numbers without error bars." That’s rule number one. What’s rule number two?

Eli: Use "paired differences" for every comparison. If you’re testing Model A versus Model B, don't just look at two tables. Compute the difference for every single prompt, find the mean of those differences, and calculate the standard error of that. That’s your "truth" for whether Model B is actually an improvement.

Nia: And if I see that those error bars for the difference include zero, I have to be honest and say, "We don't have enough evidence yet."

Eli: Exactly. That honesty saves you from "chasing noise." Rule number three: reduce variance at the source. If you have access to token probabilities, use them! Calculate the "expected score" or the log-likelihood. If you don't have probabilities, use "resampling"—ask each question multiple times and average the results.

Nia: So, we’re attacking the noise from both ends—the math we use to analyze it and the way we collect the data. What about the "small data" trap we talked about?

Eli: That’s rule number four: know your "n." If your dataset has fewer than a hundred questions, be extremely skeptical of CLT-based results. Consider using Bayesian intervals or, better yet, find a way to get more data. And always, always check for "clusters." If your questions aren't independent, use a "cluster-adjusted" standard error so you don't lie to yourself about how much evidence you really have.

Nia: I’m thinking about the "Power Analysis" too. That feels like it should be the "pre-flight check." Before you even start a big evaluation run, do a quick calculation to see if you even have enough "power" to detect the improvement you’re hoping for.

Eli: Spot on. If you’re hoping for a 2% gain but your power analysis says you need a thousand questions and you only have two hundred, you’ve just saved yourself a lot of wasted effort. You either need a bigger dataset or a more sensitive metric, like moving to continuous probabilities.

Nia: This is a total shift in mindset. It’s moving from "evals as a grade" to "evals as a measurement." It’s about knowing the precision of your instruments.

Eli: And it’s about the community too. If every researcher and every company started reporting these stats, our leaderboards would look completely different. We’d stop obsessing over 0.1% differences that are clearly within the margin of error, and we’d start focusing on the models that actually push the "error bars" forward.

Nia: I love that vision. It’s like we’re finally growing up as a field. We’re moving past the "wild west" of random numbers and into a more rigorous, scientific era. It’s exciting, honestly.

Eli: It really is. It’s the difference between guessing and knowing. And in a field that’s moving as fast as AI, knowing is the only way to keep your bearings.

Nia: Eli, this has been such an eye-opener. I feel like I can't look at a model leaderboard the same way ever again. It’s like seeing the "matrix"—once you realize how much noise is hidden in those numbers, you start looking for the statistical grounding everywhere.

Eli: It’s a bit of a "red pill" moment, right? But it’s also incredibly empowering. We don't have to be victims of noisy data. We have the tools—many of them centuries old, like the Central Limit Theorem—to tame that uncertainty and make better decisions.

Nia: I think my biggest takeaway is that "uncertainty" isn't a failure. It’s just a reality of dealing with complex, stochastic systems like LLMs. The failure is in ignoring it. When we acknowledge the error bars, we’re actually being more rigorous, not less.

Eli: You've hit the nail on the head. Science isn't about being 100% sure; it’s about knowing exactly how "not sure" you are. Whether it’s using paired differences to find that "free" variance reduction or using power analysis to plan your next benchmark, these are the habits of a mature engineering discipline.

Nia: So, to everyone listening—maybe the next time you’re looking at an evaluation result, ask yourself: "Where are the error bars?" And if they aren't there, maybe try calculating them yourself. It might just change your entire perspective on which model is actually "winning."

Eli: Absolutely. And remember, the goal isn't just a higher number—it’s a higher confidence in that number. It’s about building models that we can actually trust to perform when it matters.

Nia: I love that. Let’s stop chasing the noise and start finding the signal. It’s been a fascinating deep dive into the "math behind the magic." Thanks for walking us through it, Eli.

Eli: It’s been a blast. There’s something so satisfying about bringing a little statistical order to the chaos of LLM evals.

Nia: Truly. And for our listeners, I hope this gives you a practical playbook you can actually use. Whether you’re training a massive model or just fine-tuning a small one, these principles apply. It’s all about making progress you can actually prove.

Eli: Exactly. Happy evaluating, everyone!

Nia: Thanks for joining us today. We hope you feel a little more equipped to tackle the noise in your own work. Take a moment to reflect on your current evaluation process—is there one small step, like calculating a standard error, that you could try today? It’s a small change that leads to much bigger insights. Thanks for listening!