LLM evaluation is noisier than you think

Relying on raw scores often leads to the "highest number is best" fallacy, where tiny performance gaps are mistaken for actual progress. Research indicates that many of these decimal-point differences are simply statistical noise rather than true improvements in model capability. Without calculating statistical significance or using error bars, it is impossible to know if a model's higher score is a repeatable result or just a random fluctuation based on a specific sample of questions.

Standard deviation measures the diversity or "spread" of individual data points, showing how much the scores for different questions vary from one another. In contrast, standard error measures the precision of the average score. It tells you how much the mean performance would likely vary if you ran the same evaluation multiple times with different sets of questions. A small standard error indicates that the calculated accuracy is a reliable estimate of the model's true performance.

Paired difference analysis is a statistical "cheat code" that compares two models on the exact same set of prompts. Because models often find the same questions difficult or easy, their scores are highly correlated. By focusing on the difference in performance for each specific question rather than comparing two independent averages, the shared noise caused by question difficulty cancels out. This shrinks the standard error and allows researchers to detect significant improvements that might be hidden by the overlapping error bars of independent tests.

When a dataset has fewer than a few hundred samples, the Central Limit Theorem (CLT) may provide a false sense of security by producing confidence intervals that are too narrow. Small datasets are also prone to the "small data trap," where a model getting a perfect score (100% or 0%) makes the variance appear to be zero, incorrectly suggesting there is no uncertainty. For these smaller, specialized benchmarks, experts recommend using Bayesian methods or increasing the number of samples to ensure the results are robust.

Most evaluations use binary "pass or fail" scoring, which discards the nuance of how confident a model was in its answer. By looking at the next-token probabilities (the "expected score"), you can distinguish between a lucky guess and a confident, correct answer. This approach removes "within-question variability," leading to a much higher Signal-to-Noise Ratio (SNR). This makes performance metrics more stable and allows engineers to track progress more steadily during model training.