Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Summary and Practical Guide to Hypothesis Testing › Multiple Comparisons Reference

Michael BrenndoerferJanuary 10, 20261 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 22Summary and Practical Guide to Hypothesis Testing › Multiple Comparisons Reference

Reading section 6 of 12 · Multiple Comparisons Reference · 1 minSection 6 of 12 · 1 min

Section 6 of 12 · 1 min read

Summary and Practical Guide to Hypothesis Testing › Multiple Comparisons ReferenceOne section · Sections in this chapter6 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When to Use Each MethodLink Copied

Method	Controls	Use When
Bonferroni	FWER	Few tests, any false positive costly
Holm	FWER	Many tests, need more power than Bonferroni
Benjamini-Hochberg	FDR	Exploratory analysis, many tests OK
Tukey HSD	FWER	All pairwise comparisons after ANOVA
Dunnett	FWER	Comparing treatments to control

Quick FormulasLink Copied

Bonferroni: Reject if $p_i < \alpha/m$

Holm (step-down): For ordered p-values, reject $p_{(j)}$ if $p_{(j)} < \alpha/(m-j+1)$

Benjamini-Hochberg: Find largest $k$ where $p_{(k)} \leq \frac{k}{m}q$ , reject all $p \leq p_{(k)}$

In[6]:

Code

from scipy.stats import false_discovery_control

# Example: 5 p-values from multiple tests
p_values = [0.001, 0.008, 0.039, 0.041, 0.23]

# Bonferroni
bonf_threshold = 0.05 / len(p_values)
bonf_reject = [p < bonf_threshold for p in p_values]
print(f"Bonferroni (threshold = {bonf_threshold:.3f}): {bonf_reject}")

# Benjamini-Hochberg
bh_adjusted = false_discovery_control(p_values, method="bh")
bh_reject = bh_adjusted < 0.05
print(f"BH adjusted p-values: {[f'{p:.3f}' for p in bh_adjusted]}")
print(f"BH reject: {list(bh_reject)}")

Out[6]:

Console

Bonferroni (threshold = 0.010): [True, True, False, False, False]
BH adjusted p-values: ['0.005', '0.020', '0.051', '0.051', '0.230']
BH reject: [np.True_, np.True_, np.False_, np.False_, np.False_]