Multiple Comparisons › The Multiple Comparisons Problem

When we conduct a single hypothesis test at significance level $\alpha = 0.05$, we accept a 5% chance of a Type I error: rejecting a true null hypothesis. This seems like an acceptable risk. But what happens when we conduct many tests?

Consider conducting $m$ independent hypothesis tests, each at significance level $\alpha$. If all null hypotheses are true, each test has probability $\alpha$ of producing a false positive. The probability of making at least one false positive across all tests is:

$$P(\text{at least one false positive}) = 1 - P(\text{no false positives})$$

Since the tests are independent:

$$P(\text{no false positives}) = (1 - \alpha)^m$$

Therefore:

$$P(\text{at least one false positive}) = 1 - (1 - \alpha)^m$$

This is called the family-wise error rate (FWER): the probability of making at least one Type I error among a family of tests.

Let's see how quickly this grows:

With just 14 tests, you have more than a 50% chance of at least one false positive. By 100 tests, a false positive is virtually certain (99.4%).

Let's simulate the multiple comparisons problem directly. We'll generate data where all null hypotheses are true (no real effects) and watch how false positives accumulate.