Multiple Comparisons › Summary
Machine Learning from ScratchAll chapters Multiple Comparisons › SummaryOne section · Sections in this chapter10 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The multiple comparisons problem is one of the most important concepts in modern statistics. When conducting many hypothesis tests, the probability of false positives increases dramatically: from 5% for a single test to 99.4% for 100 tests at .
Key concepts:
- Family-wise error rate (FWER): Probability of at least one false positive; controlled by Bonferroni, Šidák, Holm, Hochberg
- False discovery rate (FDR): Expected proportion of false positives among rejections; controlled by Benjamini-Hochberg, Benjamini-Yekutieli
- FWER methods are more conservative, appropriate for confirmatory research
- FDR methods allow more discoveries, appropriate for exploratory research
Correction methods:
| Method | Controls | Power | Assumptions |
|---|---|---|---|
| Bonferroni | FWER | Low | None |
| Šidák | FWER | Low-Moderate | Independence |
| Holm | FWER | Moderate | None |
| Hochberg | FWER | Moderate-High | Independence/positive dependence |
| BH | FDR | High | Independence/positive dependence |
| BY | FDR | Moderate | None |
For ANOVA post-hoc tests:
- Tukey HSD for all pairwise comparisons
- Dunnett for comparisons with control
- Scheffé for complex contrasts
The dead salmon study reminds us that even absurd results can appear "significant" when we conduct enough tests without correction. Proper handling of multiple comparisons is essential for valid statistical inference.