Why is it problematic to rely on raw accuracy scores in LLM leaderboards?

Relying on raw scores often leads to the "highest number is best" fallacy, where tiny performance gaps are mistaken for actual progress. Research indicates that many of these decimal-point differences are simply statistical noise rather than true improvements in model capability. Without calculating statistical significance or using error bars, it is impossible to know if a model's higher score is a repeatable result or just a random fluctuation based on a specific sample of questions.

What is the difference between standard deviation and standard error in the context of model evaluation?

Standard deviation measures the diversity or "spread" of individual data points, showing how much the scores for different questions vary from one another. In contrast, standard error measures the precision of the average score. It tells you how much the mean performance would likely vary if you ran the same evaluation multiple times with different sets of questions. A small standard error indicates that the calculated accuracy is a reliable estimate of the model's true performance.

How can "paired differences" make model comparisons more accurate?

Paired difference analysis is a statistical "cheat code" that compares two models on the exact same set of prompts. Because models often find the same questions difficult or easy, their scores are highly correlated. By focusing on the difference in performance for each specific question rather than comparing two independent averages, the shared noise caused by question difficulty cancels out. This shrinks the standard error and allows researchers to detect significant improvements that might be hidden by the overlapping error bars of independent tests.

Why should researchers be cautious when using small datasets for evaluations?

When a dataset has fewer than a few hundred samples, the Central Limit Theorem (CLT) may provide a false sense of security by producing confidence intervals that are too narrow. Small datasets are also prone to the "small data trap," where a model getting a perfect score (100% or 0%) makes the variance appear to be zero, incorrectly suggesting there is no uncertainty. For these smaller, specialized benchmarks, experts recommend using Bayesian methods or increasing the number of samples to ensure the results are robust.

How does using token probabilities reduce noise in evaluation results?

Most evaluations use binary "pass or fail" scoring, which discards the nuance of how confident a model was in its answer. By looking at the next-token probabilities (the "expected score"), you can distinguish between a lucky guess and a confident, correct answer. This approach removes "within-question variability," leading to a much higher Signal-to-Noise Ratio (SNR). This makes performance metrics more stable and allows engineers to track progress more steadily during model training.

LLM evaluation is noisier than you think

28 min

Mar 31, 2026

AI Science Technology

Leaderboard rankings often mistake noise for progress. Learn how to use statistical tools to find real signals and build more reliable model benchmarks.

Best quote from LLM evaluation is noisier than you think

Science isn't about being 100% sure; it’s about knowing exactly how 'not sure' you are. When we acknowledge the error bars, we’re actually being more rigorous, not less.

This audio lesson was created by a BeFreed community member

Input question

https://cameronrwolfe.substack.com/p/stats-llm-evals

Host voices

Nia

Eli

Learning style

Deep

Knowledge sources

Direct source: cameronrwolfe.substack.com

https://cameronrwolfe.substack.com/p/stats-llm-evals

Frequently Asked Questions

From Columbia University alumni built in San Francisco

BeFreed Brings Together A Global Community Of 1,000,000 Curious Minds

See more on how BeFreed is discussed across the web

"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."

@Moemenn

"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."

@Chloe, Solo founder, LA

117

"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."

@Raaaaaachelw

"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."

@Matt, YC alum

108

"Reading used to feel like a chore. Now it’s just part of my lifestyle."

@Erin, Investment Banking Associate , NYC

254

"Feels effortless compared to reading. I’ve finished 6 books this month already."

@djmikemoore

"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."

@Pitiful

4.5K

"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."

@SofiaP

"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"

@Jaded_Falcon

201

"It is great for me to learn something from the book without reading it."

@OojasSalunke

"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."

@Leo, Law Student, UPenn

483

"Makes me feel smarter every time before going to work"

@Cashflowbubu

From Columbia University alumni built in San Francisco

BeFreed Brings Together A Global Community Of 1,000,000 Curious Minds

See more on how BeFreed is discussed across the web

"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."

@Moemenn

"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."

@Chloe, Solo founder, LA

117

"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."

@Raaaaaachelw

"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."

@Matt, YC alum

108

"Reading used to feel like a chore. Now it’s just part of my lifestyle."

@Erin, Investment Banking Associate , NYC

254

"Feels effortless compared to reading. I’ve finished 6 books this month already."

@djmikemoore

"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."

@Pitiful

4.5K

"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."

@SofiaP

"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"

@Jaded_Falcon

201

"It is great for me to learn something from the book without reading it."

@OojasSalunke

"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."

@Leo, Law Student, UPenn

483

"Makes me feel smarter every time before going to work"

@Cashflowbubu

"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."

@Moemenn

"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."

@Chloe, Solo founder, LA

117

"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."

@Raaaaaachelw

"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."

@Matt, YC alum

108

"Reading used to feel like a chore. Now it’s just part of my lifestyle."

@Erin, Investment Banking Associate , NYC

254

"Feels effortless compared to reading. I’ve finished 6 books this month already."

@djmikemoore

"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."

@Pitiful

4.5K

"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."

@SofiaP

"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"

@Jaded_Falcon

201

"It is great for me to learn something from the book without reading it."

@OojasSalunke

"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."

@Leo, Law Student, UPenn

483

"Makes me feel smarter every time before going to work"

@Cashflowbubu

"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."

@Moemenn

"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."

@Chloe, Solo founder, LA

117

"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."

@Raaaaaachelw

"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."

@Matt, YC alum

108

"Reading used to feel like a chore. Now it’s just part of my lifestyle."

@Erin, Investment Banking Associate , NYC

254

"Feels effortless compared to reading. I’ve finished 6 books this month already."

@djmikemoore

"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."

@Pitiful

4.5K

"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."

@SofiaP

"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"

@Jaded_Falcon

201

"It is great for me to learn something from the book without reading it."

@OojasSalunke

"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."

@Leo, Law Student, UPenn

483

"Makes me feel smarter every time before going to work"

@Cashflowbubu

1.5K Ratings4.7

Start your learning journey, now

LLM evaluation is noisier than you think

28 min

Mar 31, 2026

AI Science Technology

Leaderboard rankings often mistake noise for progress. Learn how to use statistical tools to find real signals and build more reliable model benchmarks.

Best quote from LLM evaluation is noisier than you think

Science isn't about being 100% sure; it’s about knowing exactly how 'not sure' you are. When we acknowledge the error bars, we’re actually being more rigorous, not less.