Multiple Comparisons › Error Rate Concepts: FWER vs FDR

Before diving into correction methods, we need to clearly distinguish two different ways to conceptualize error rates in multiple testing. Each has its place depending on your research goals.

The family-wise error rate is the probability of making at least one Type I error among all tests:

$$\text{FWER} = P(V \geq 1)$$

where $V$ is the number of false positives (true null hypotheses incorrectly rejected).

When to control FWER:

• Confirmatory research where conclusions must be definitive
• Clinical trials where each below-threshold p-value can lead to treatment decisions
• Any situation where a single false positive has serious consequences
• Small numbers of pre-specified comparisons

Controlling FWER at level $\alpha$ guarantees that the probability of any false positive is at most $\alpha$. This is the most conservative form of error control.

The false discovery rate is the expected proportion of false positives among rejected hypotheses:

$$\text{FDR} = E\left[\frac{V}{R}\right] = E\left[\frac{V}{\max(R, 1)}\right]$$

where $V$ is the number of false positives and $R$ is the total number of rejections.

When to control FDR:

• Exploratory research where you're screening many hypotheses
• Genomics and proteomics, including other -omics studies
• Neuroimaging with thousands of voxels
• A/B testing with many metrics
• Situations where some false positives are acceptable if most discoveries are real

Controlling FDR at level $q$ means that among your "significant" results, the expected proportion of false discoveries is at most $q$. This allows more discoveries while still controlling the error rate.