Why LLM Leaderboards Are Often Wrong
Small score gaps in model evals might just be noise. Learn how to use statistical error bars and rigor to determine if your model is actually better.
Best quote from Why LLM Leaderboards Are Often Wrong
The biggest red flag in AI right now isn't a low score—it’s a high score with no error bars. We need to stop treating evals like static scores and start treating them like the scientific experiments they actually are.
This audio lesson was created by a BeFreed community member
Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations Evan Miller Anthropic evanmiller@anthropic.com Abstract Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning. This article shows researchers with some training in statistics how to think about and analyze...
Frequently Asked Questions
Ranking models by tiny margins—such as a 0.5% difference—is often misleading because these fluctuations may simply be statistical noise rather than a reflection of true capability. Evaluation datasets are finite samples pulled from a theoretical "super-population" of all possible questions. Without calculating error bars or standard error, it is impossible to know if a higher score is a significant result or if the ranks would flip if the experiment were run again with different questions or different model seeds.
The Rule of Three is a statistical guideline used when a model passes every single test in a small sample size. If you run 30 safety tests and the model never fails, it is mathematically incorrect to claim the model is 100% safe. Instead, the rule dictates that the 95% confidence upper bound for the failure rate is 3 divided by the number of tests. In a 30-test scenario, you can only say with 95% confidence that the failure rate is below 10% in the wild.
Standard statistical assumptions require that every question in a dataset be independent, but real-world benchmarks often violate this by using multiple questions based on the same document or translating the same prompt into different languages. If a model struggles with the underlying context, it will likely fail all related questions, meaning they are not independent "votes" on performance. Clustered Standard Errors account for this correlation by grouping related items, preventing researchers from underestimating uncertainty and reporting artificially small error bars.
One of the most effective ways to shrink error bars is to use continuous metrics like "logprobs" (log probabilities) instead of binary pass/fail scores. By looking at the probability the model assigned to the correct answer rather than whether it happened to sample that answer, you eliminate "within-question" variance caused by the model's internal randomness. Other strategies include resampling (averaging multiple completions for the same prompt) and averaging results across the final few checkpoints of a training run to smooth out lucky fluctuations in model weights.
Comparing two separate error bars is often too conservative; models can have overlapping confidence intervals and still show a statistically significant difference. A paired difference test evaluates both models on the exact same set of questions and focuses on the gap between their scores. Because models usually agree on which questions are difficult, their scores are positively correlated. Subtracting these correlated variables shrinks the variance of the difference, making the test much more sensitive and capable of detecting real improvements that a naive comparison would miss.
Discover more
Python programming for LLMs and evals
As AI integration becomes standard, the ability to both build and critically evaluate models is a vital technical differentiator. This path is ideal for developers and data scientists looking to transition from general programming to specialized LLM engineering and rigorous model benchmarking.
LLM Training: From Raw Text to Aligned Assistant
As the demand for custom AI grows, understanding the full lifecycle of model development is essential for engineers. This plan is ideal for data scientists and systems engineers looking to bridge the gap between raw data engineering and advanced model alignment at scale.
AI Myths: LLMs vs. True Sentience
This learning plan is essential for anyone looking to look past the headlines and understand the actual capabilities of modern AI. It is particularly valuable for tech enthusiasts, students, and professionals who want to ground their understanding of machine intelligence in both science and philosophy.
large language models
As AI reshapes industries, understanding the mechanics of large language models is essential for developers and researchers. This plan bridges the gap between theoretical mathematics and practical deployment, making it ideal for those looking to build responsible and powerful AI systems.
LLM personalization and memory
This learning plan is essential for AI engineers, ML practitioners, and developers who want to move beyond basic LLM usage to create truly intelligent, personalized applications. As businesses demand AI systems that understand context, remember user preferences, and adapt over time, the ability to implement memory systems and personalization techniques has become a critical competitive advantage in the AI space.
LSAT Logic & Games Mastery Foundation Program
This program is essential for law school applicants who need to transform their approach to the LSAT from intuition-based to strategy-driven. It bridges the gap between basic logic and the high-speed analytical demands of the actual exam, making it ideal for anyone aiming for a top-tier score.
AI Decision Models: Constraints & Failures
As AI systems increasingly make consequential decisions in healthcare, finance, and public safety, understanding their limitations becomes critical. This plan equips professionals and decision-makers with the knowledge to evaluate AI systems realistically and build more reliable models that avoid common pitfalls.
大模型数据极限下的技术博弈
随着互联网公开数据趋于极限，如何突破数据瓶颈已成为大模型竞争的核心。本课程深入探讨合成数据、数据策展及算法优化，适合希望掌握前沿大模型训练策略的算法工程师与技术决策者。
From Columbia University alumni built in San Francisco
"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."
"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."
"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."
"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."
"Reading used to feel like a chore. Now it’s just part of my lifestyle."
"Feels effortless compared to reading. I’ve finished 6 books this month already."
"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."
"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."
"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"
"It is great for me to learn something from the book without reading it."
"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."
"Makes me feel smarter every time before going to work"
From Columbia University alumni built in San Francisco
"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."
"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."
"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."
"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."
"Reading used to feel like a chore. Now it’s just part of my lifestyle."
"Feels effortless compared to reading. I’ve finished 6 books this month already."
"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."
"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."
"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"
"It is great for me to learn something from the book without reading it."
"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."
"Makes me feel smarter every time before going to work"
"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."
"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."
"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."
"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."
"Reading used to feel like a chore. Now it’s just part of my lifestyle."
"Feels effortless compared to reading. I’ve finished 6 books this month already."
"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."
"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."
"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"
"It is great for me to learn something from the book without reading it."
"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."
"Makes me feel smarter every time before going to work"
"Instead of endless scrolling, I just hit play on BeFreed. It saves me so much time."
"I never knew where to start with nonfiction—BeFreed’s book lists turned into podcasts gave me a clear path."
"Perfect balance between learning and entertainment. Finished ‘Thinking, Fast and Slow’ on my commute this week."
"Crazy how much I learned while walking the dog. BeFreed = small habits → big gains."
"Reading used to feel like a chore. Now it’s just part of my lifestyle."
"Feels effortless compared to reading. I’ve finished 6 books this month already."
"BeFreed turned my guilty doomscrolling into something that feels productive and inspiring."
"BeFreed turned my commute into learning time. 20-min podcasts are perfect for finishing books I never had time for."
"BeFreed replaced my podcast queue. Imagine Spotify for books — that’s it. 🙌"
"It is great for me to learn something from the book without reading it."
"The themed book list podcasts help me connect ideas across authors—like a guided audio journey."
"Makes me feel smarter every time before going to work"
![[2411.00640] Adding Error Bars to Evals: A Statistical Approach to ...](/_next/image?url=https%3A%2F%2Fd1y2du6z1jfm9e.cloudfront.net%2Fassets%2Fpodcast%2Fyellow.png&w=1024&q=50)
