Type I and Type II Errors › Type I Errors: The False Alarm
Machine Learning from ScratchAll chapters Type I and Type II Errors › Type I Errors: The False AlarmOne section · Sections in this chapter3 / 10 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
A Type I error occurs when you reject a true null hypothesis. In plain language: you conclude that something interesting is happening when, in reality, nothing is going on. You've raised a false alarm.
The Probability of Type I Error: α
The probability of a Type I error is denoted by the Greek letter alpha (α), and it equals the significance level you choose for your test. When you set α = 0.05, you are accepting a 5% probability of falsely rejecting the null hypothesis when it is true.
Mathematically:
This is a conditional probability: the probability of rejecting the null hypothesis, given that the null hypothesis is true. It represents the false positive rate of your test.
Why α Equals the Significance Level
To understand why α equals our chosen significance level, recall how hypothesis testing works. We:
- Assume the null hypothesis is true
- Calculate the sampling distribution of our test statistic under this assumption
- Determine what values of the test statistic would be "extreme enough" to reject
- The significance level is precisely the probability of observing such extreme values when is true
For a two-tailed z-test with α = 0.05:
The Key Insight: You Control α
Unlike many other aspects of hypothesis testing, the significance level α is under your direct control. You choose it before conducting the test, based on the consequences of false positives in your specific context.
The conventional choice of α = 0.05 is just that, a convention. It was popularized by Ronald Fisher in the early 20th century as a reasonable default, but it is not a universal law. Different contexts warrant different choices:
| Context | Typical α | Rationale |
|---|---|---|
| Exploratory research | 0.10 | Missing effects is costly; expect replication |
| Standard scientific research | 0.05 | Convention balancing Type I and II errors |
| Confirmatory/regulatory | 0.01 | False positives have serious consequences |
| Particle physics discoveries | ~0.0000003 (5σ) | Extraordinary claims require extraordinary evidence |
| Genome-wide association | 5 × 10⁻⁸ | Multiple testing across millions of variants |
Real-World Consequences of Type I Errors
Type I errors can have serious consequences across many domains:
Medical diagnosis: A healthy patient is told they have cancer. This causes severe psychological distress, leads to invasive follow-up procedures (biopsies, additional imaging), and may result in unnecessary treatment with harmful side effects.
Criminal justice: An innocent person is convicted of a crime. This is the scenario our legal systems are designed to prevent: the presumption of innocence exists precisely because Type I errors (convicting the innocent) are considered worse than Type II errors (acquitting the guilty).
Drug approval: The FDA approves a drug that is no better than placebo. Patients take an ineffective medication, potentially experiencing side effects without any benefit, while being denied treatments that might work.
A/B testing: A company deploys a new website design based on a "significant" result that was just noise. Engineering resources are wasted, and if the change is harmful, user experience suffers.
Scientific research: A researcher publishes a "discovery" that is just a statistical fluke. Other researchers waste time and resources trying to replicate or build on the finding. The scientific literature becomes polluted with false results.