ANOVA (Analysis of Variance) › Post-Hoc Tests: Which Groups Differ?
Machine Learning from ScratchAll chapters ANOVA (Analysis of Variance) › Post-Hoc Tests: Which Groups Differ?One section · Sections in this chapter6 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When ANOVA rejects the null hypothesis, we know at least one group mean differs from the others, but not which specific pairs differ. Post-hoc tests make pairwise comparisons while controlling the family-wise error rate.
The Need for Multiple Comparison Corrections
Without correction, testing all pairwise differences would inflate the Type I error rate, exactly the problem ANOVA was designed to solve. Post-hoc tests apply corrections to maintain the overall error rate at .
Tukey's Honest Significant Difference (HSD)
Tukey's HSD is the most commonly used post-hoc test for ANOVA. It controls the family-wise error rate while comparing all pairs of means.
The test compares the difference between any two group means to a critical value based on the studentized range distribution:
where is the critical value from the studentized range distribution.
Two means are significantly different if .
from itertools import combinations
from scipy import stats
# Perform pairwise comparisons with Bonferroni correction
# (conservative alternative when studentized range tables unavailable)
groups = [fertilizer_a, fertilizer_b, fertilizer_c]
group_names = ["Fertilizer A", "Fertilizer B", "Fertilizer C"]
# Calculate pooled within-group variance (MSW)
k = len(groups)
N = sum(len(g) for g in groups)
means = [np.mean(g) for g in groups]
ss_within = sum(np.sum((g - m) ** 2) for g, m in zip(groups, means))
ms_within = ss_within / (N - k)
print("Post-Hoc Pairwise Comparisons (Bonferroni-corrected)")
print("=" * 65)
print()
n_comparisons = k * (k - 1) // 2
alpha_corrected = 0.05 / n_comparisons
for (i, gi), (j, gj) in combinations(enumerate(groups), 2):
mean_diff = means[i] - means[j]
ni, nj = len(gi), len(gj)
se = np.sqrt(ms_within * (1 / ni + 1 / nj))
t_stat = mean_diff / se
df = N - k
p_raw = 2 * stats.t.sf(abs(t_stat), df)
p_corrected = min(p_raw * n_comparisons, 1.0)
sig = "*" if p_corrected < 0.05 else ""
print(f"{group_names[i]} vs {group_names[j]}:")
print(f" Mean difference: {mean_diff:+.2f} cm")
print(f" p-value (corrected): {p_corrected:.4f} {sig}")
print()
print(
"* indicates significant difference at alpha = 0.05 (Bonferroni-corrected)"
)Post-Hoc Pairwise Comparisons (Bonferroni-corrected) ================================================================= Fertilizer A vs Fertilizer B: Mean difference: -6.00 cm p-value (corrected): 0.0000 * Fertilizer A vs Fertilizer C: Mean difference: -2.30 cm p-value (corrected): 0.0058 * Fertilizer B vs Fertilizer C: Mean difference: +3.70 cm p-value (corrected): 0.0000 * * indicates significant difference at alpha = 0.05 (Bonferroni-corrected)
The post-hoc analysis reveals:
- Fertilizer B produces significantly more growth than both A and C
- Fertilizers A and C do not differ significantly from each other
This is consistent with the visual impression from the boxplots: Fertilizer B is clearly the best performer.