ANOVA (Analysis of Variance) › The Multiple Comparisons Problem

Michael BrenndoerferJanuary 5, 20263 min read
Machine Learning from ScratchAll chapters
Part 2 of 9Hypothesis Testing
Chapter 17ANOVA (Analysis of Variance) › The Multiple Comparisons Problem
Reading section 2 of 13 · The Multiple Comparisons Problem · 3 minSection 2 of 13 · 3 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 17 of 66

Section 2 of 13 · 3 min read

ANOVA (Analysis of Variance) › The Multiple Comparisons ProblemOne section · Sections in this chapter2 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before diving into ANOVA, let's understand exactly why multiple t-tests fail.

When you conduct a single hypothesis test at α=0.05\alpha = 0.05, you accept a 5% chance of a Type I error (false positive) when the null is true. But with multiple tests, each test adds another opportunity for error.

The probability of at least one false positive across mm independent tests is:

P(at least one Type I error)=1−(1−α)mP(\text{at least one Type I error}) = 1 - (1 - \alpha)^m

For α=0.05\alpha = 0.05:

Number of GroupsPairwise ComparisonsP(At least one Type I error)
3314.3%
4626.5%
51040.1%
104590.1%

This explosion of false positives makes multiple t-tests unreliable. ANOVA solves this by asking a single question: "Do any of the group means differ?" rather than asking many questions about specific pairs.

Out[2]:
Visualization
Line plot showing false positive rate increasing with number of groups.
The multiple comparisons problem: as the number of groups increases, the probability of at least one false positive rises dramatically when using separate pairwise tests. With 10 groups (45 comparisons), you''d expect to find at least one ''significant'' difference over 90% of the time, even when all groups are identical.