Multiple Comparisons › Common Pitfalls and Best Practices

Michael BrenndoerferJanuary 9, 20262 min read
Machine Learning from ScratchAll chapters
Part 2 of 9Hypothesis Testing
Chapter 21Multiple Comparisons › Common Pitfalls and Best Practices
Reading section 8 of 12 · Common Pitfalls and Best Practices · 2 minSection 8 of 12 · 2 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 21 of 66

Section 8 of 12 · 2 min read

Multiple Comparisons › Common Pitfalls and Best PracticesOne section · Sections in this chapter8 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Pitfall 1: Ignoring Multiple Comparisons

The most serious pitfall is ignoring the multiple comparisons problem entirely. This leads to inflated false positive rates and unreliable conclusions.

Warning: The Hidden Multiplicity

Multiple comparisons can be hidden in various ways:

  • Testing many subgroups without pre-specification
  • Analyzing data at multiple time points
  • Testing multiple outcomes without correction
  • "Peeking" at data during sequential data collection
  • Testing different model specifications

All of these constitute multiple testing and require appropriate correction.

Pitfall 2: Overcorrecting

Applying overly conservative corrections (like Bonferroni when BH would suffice) reduces power and may cause you to miss real effects. Match your correction method to your research goals.

Pitfall 3: Correcting Within Correlated Families

When tests are strongly positively correlated (e.g., testing the same hypothesis in overlapping subsamples), standard corrections become overly conservative. Consider methods designed for dependent tests or reduce to independent tests.

Best Practices

  1. Pre-specify your analysis plan: Define which tests you'll conduct before seeing the data
  2. Match correction to research phase: Use FWER for confirmatory research, FDR for exploratory
  3. Report the correction method used: Always state which correction you applied
  4. Report both raw and adjusted p-values: Allows readers to apply their own standards
  5. Consider the dependence structure: Use BH for independent/positive dependence, BY for arbitrary dependence
  6. Focus on effect sizes: A barely significant corrected p-value with a tiny effect size isn't very informative