Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multiple Comparisons › Post-Hoc Tests for ANOVA

Michael BrenndoerferJanuary 9, 20263 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 21Multiple Comparisons › Post-Hoc Tests for ANOVA

Reading section 6 of 12 · Post-Hoc Tests for ANOVA · 3 minSection 6 of 12 · 3 min

Section 6 of 12 · 3 min read

Multiple Comparisons › Post-Hoc Tests for ANOVAOne section · Sections in this chapter6 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When ANOVA reveals significant differences among groups, post-hoc tests determine which specific pairs differ. These are inherently multiple comparison problems since comparing $k$ groups involves $\binom{k}{2} = \frac{k(k-1)}{2}$ pairwise comparisons.

Tukey's Honest Significant Difference (HSD)Link Copied

Tukey's HSD is the most common post-hoc test for ANOVA. It controls the family-wise error rate for all pairwise comparisons when group sizes are equal.

The method: Two groups $i$ and $j$ are significantly different if:

|\bar{x}_i - \bar{x}_j| > q_{\alpha, k, df_W} \cdot \sqrt{\frac{MS_W}{n}}

where:

$q_{\alpha, k, df_W}$ is the critical value from the Studentized Range distribution
$k$ is the number of groups
$df_W = N - k$ is the within-groups degrees of freedom
$MS_W$ is the within-groups mean square (from ANOVA)
$n$ is the sample size per group

When to use:

All pairwise comparisons are of interest
Equal (or approximately equal) sample sizes
Homogeneity of variance assumption met

In[13]:

Code

from scipy.stats import tukey_hsd

# Example: Three teaching methods
np.random.seed(42)
method_a = np.random.normal(75, 10, 25)  # Traditional
method_b = np.random.normal(82, 10, 25)  # Interactive
method_c = np.random.normal(78, 10, 25)  # Mixed

# Perform Tukey's HSD
result = tukey_hsd(method_a, method_b, method_c)
print(result)

Out[13]:

Console

Pairwise Group Comparisons (95.0% Confidence Interval)
Comparison  Statistic  p-value  Lower CI  Upper CI
 (0 - 1)     -5.761     0.091   -12.237     0.715
 (0 - 2)     -5.696     0.096   -12.172     0.780
 (1 - 0)      5.761     0.091    -0.715    12.237
 (1 - 2)      0.065     1.000    -6.411     6.541
 (2 - 0)      5.696     0.096    -0.780    12.172
 (2 - 1)     -0.065     1.000    -6.541     6.411

Dunnett's TestLink Copied

Dunnett's test compares each treatment group to a single control group. It's more powerful than Tukey's HSD when you only care about comparisons with the control.

When to use:

One group is a control/reference
Only comparisons with control are of interest
More powerful than Tukey for this specific situation

In[14]:

Code

from scipy.stats import dunnett

# Example: Control vs two treatments
np.random.seed(42)
control = np.random.normal(50, 8, 20)
treatment1 = np.random.normal(55, 8, 20)
treatment2 = np.random.normal(58, 8, 20)

# Dunnett's test (comparing each treatment to control)
result = dunnett(treatment1, treatment2, control=control)
print(f"Treatment 1 vs Control: p = {result.pvalue[0]:.4f}")
print(f"Treatment 2 vs Control: p = {result.pvalue[1]:.4f}")

Out[14]:

Console

Treatment 1 vs Control: p = 0.1300
Treatment 2 vs Control: p = 0.0004

Scheffé's MethodLink Copied

Scheffé's method is the most flexible but also most conservative post-hoc test. It controls FWER for all possible contrasts (not just pairwise comparisons), including complex comparisons like "Group A vs. the average of Groups B and C."

When to use:

Unplanned (post-hoc) comparisons
Complex contrasts beyond simple pairwise comparisons
Unequal sample sizes
Most conservative option

Summary of Post-Hoc MethodsLink Copied

Method	Comparisons	Sample Sizes	Power	Use Case
Tukey HSD	All pairwise	Equal	Moderate	Standard choice for all pairwise
Bonferroni	Any	Any	Low	Simple, always valid
Holm	Any	Any	Moderate	Better than Bonferroni
Dunnett	vs. Control only	Any	High	Treatment vs. control
Scheffé	Any contrast	Any	Low	Complex, unplanned contrasts