Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multiple Comparisons › FWER-Controlling Methods

Michael BrenndoerferJanuary 9, 20266 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 21Multiple Comparisons › FWER-Controlling Methods

Reading section 4 of 12 · FWER-Controlling Methods · 6 minSection 4 of 12 · 6 min

Section 4 of 12 · 6 min read

Multiple Comparisons › FWER-Controlling MethodsOne section · Sections in this chapter4 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Bonferroni CorrectionLink Copied

The Bonferroni correction is the simplest and most widely known method for controlling FWER. The idea is mathematically elegant: if you want the overall error rate to be at most $\alpha$ across $m$ tests, make each individual test more stringent.

The method: To test $m$ hypotheses while controlling FWER at level $\alpha$ :

Reject hypothesis $i$ if $p_i < \alpha/m$
Equivalently, compute adjusted p-values: $\tilde{p}_i = \min(m \cdot p_i, 1)$

Mathematical justification: The Bonferroni inequality states that for any events $A_1, \ldots, A_m$ :

P\left(\bigcup_{i=1}^{m} A_i\right) \leq \sum_{i=1}^{m} P(A_i)

Let $A_i$ be the event that test $i$ produces a false positive. If we use threshold $\alpha/m$ for each test:

\text{FWER} = P\left(\bigcup_{i=1}^{m} A_i\right) \leq \sum_{i=1}^{m} P(A_i) = \sum_{i=1}^{m} \frac{\alpha}{m} = \alpha

Properties:

Always valid (makes no assumptions about test dependence)
Very conservative, especially for large $m$
Power decreases as $m$ increases
Simple to implement and explain

In[6]:

Code

def bonferroni_correction(p_values, alpha=0.05):
    """
    Apply Bonferroni correction for multiple comparisons.

    Parameters
    ----------
    p_values : array-like
        Raw p-values from multiple tests
    alpha : float
        Desired FWER level

    Returns
    -------
    adjusted_p : ndarray
        Bonferroni-adjusted p-values
    rejected : ndarray
        Boolean array indicating which hypotheses to reject
    """
    p = np.asarray(p_values)
    m = len(p)
    adjusted_p = np.minimum(p * m, 1.0)
    rejected = p < (alpha / m)
    return adjusted_p, rejected

Šidák CorrectionLink Copied

The Šidák correction is slightly less conservative than Bonferroni and is exact when tests are independent.

The method: For $m$ independent tests at FWER level $\alpha$ :

Reject hypothesis $i$ if $p_i < 1 - (1-\alpha)^{1/m}$

Mathematical justification: If tests are independent and we want FWER = $\alpha$ :

\text{FWER} = 1 - (1-\alpha_{\text{per-test}})^m = \alpha

Solving for the per-test significance level:

\alpha_{\text{per-test}} = 1 - (1-\alpha)^{1/m}

Comparison with Bonferroni: For $m = 20$ tests at $\alpha = 0.05$ :

Bonferroni: $\alpha/m = 0.0025$
Šidák: $1 - (1-0.05)^{1/20} = 0.00256$

The difference is small but Šidák is slightly more powerful. However, Šidák requires independence while Bonferroni is always valid.

In[7]:

Code

def sidak_correction(p_values, alpha=0.05):
    """
    Apply Šidák correction for multiple comparisons.
    Requires test independence for exact FWER control.
    """
    p = np.asarray(p_values)
    m = len(p)
    alpha_adj = 1 - (1 - alpha) ** (1 / m)
    # Adjusted p-values
    adjusted_p = 1 - (1 - p) ** m
    rejected = p < alpha_adj
    return adjusted_p, rejected

Holm's Step-Down ProcedureLink Copied

Holm's method (1979) is uniformly more powerful than Bonferroni while still controlling FWER. The key insight is that we can be less stringent on tests with larger p-values.

The algorithm:

Order p-values: $p_{(1)} \leq p_{(2)} \leq \ldots \leq p_{(m)}$
For $j = 1, 2, \ldots, m$ :
- If $p_{(j)} > \alpha/(m-j+1)$ , stop and reject all hypotheses with smaller p-values
- Otherwise, continue to the next p-value
If all p-values pass their thresholds, reject all hypotheses

Mathematical justification: The adjusted thresholds form a step-down sequence:

\frac{\alpha}{m}, \frac{\alpha}{m-1}, \frac{\alpha}{m-2}, \ldots, \frac{\alpha}{2}, \alpha

The first comparison uses the Bonferroni threshold, but subsequent comparisons are less stringent. This maintains FWER control because once we fail to reject at step $j$ , we know that hypothesis has $p > \alpha/(m-j+1)$ , and remaining hypotheses have even larger p-values.

In[8]:

Code

def holm_correction(p_values, alpha=0.05):
    """
    Apply Holm's step-down procedure for FWER control.

    More powerful than Bonferroni while maintaining FWER control.
    """
    p = np.asarray(p_values)
    m = len(p)

    # Get sorted indices
    sorted_idx = np.argsort(p)
    p_sorted = p[sorted_idx]

    # Calculate Holm thresholds
    thresholds = alpha / (m - np.arange(m))

    # Find first non-significant result
    rejected_sorted = np.zeros(m, dtype=bool)
    for j in range(m):
        if p_sorted[j] > thresholds[j]:
            break
        rejected_sorted[j] = True

    # Map back to original order
    rejected = np.zeros(m, dtype=bool)
    rejected[sorted_idx] = rejected_sorted

    # Calculate adjusted p-values
    adjusted_sorted = np.zeros(m)
    for j in range(m):
        adjusted_sorted[j] = min(
            max(p_sorted[: j + 1] * (m - np.arange(j + 1))), 1.0
        )
    adjusted_p = np.zeros(m)
    adjusted_p[sorted_idx] = adjusted_sorted

    return adjusted_p, rejected

Hochberg's Step-Up ProcedureLink Copied

Hochberg's method (1988) is similar to Holm but works in the opposite direction (step-up instead of step-down). It's more powerful than Holm but requires the assumption that tests are independent or positively correlated.

The algorithm:

Order p-values: $p_{(1)} \leq p_{(2)} \leq \ldots \leq p_{(m)}$
For $j = m, m-1, \ldots, 1$ (starting from largest):
- If $p_{(j)} \leq \alpha/(m-j+1)$ , reject all hypotheses with p-values $\leq p_{(j)}$
- Otherwise, continue to the next smaller p-value

When to use:

When tests are independent or positively correlated
When you want more power than Holm
Not valid for negatively correlated tests

Comparing FWER MethodsLink Copied

Out[9]:

Visualization