Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Type I and Type II Errors › The Fundamental Tradeoff

Michael BrenndoerferJanuary 6, 20264 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 18Type I and Type II Errors › The Fundamental Tradeoff

Reading section 6 of 10 · The Fundamental Tradeoff · 4 minSection 6 of 10 · 4 min

Section 6 of 10 · 4 min read

Type I and Type II Errors › The Fundamental TradeoffOne section · Sections in this chapter6 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

For a fixed sample size, there is an inevitable tradeoff between Type I and Type II errors. Decreasing α (being more stringent about false positives) necessarily increases β (making false negatives more likely), and vice versa.

This tradeoff is best visualized by showing both distributions, the null and the alternative, and seeing how the critical value divides the space.

Out[9]:

Visualization

Overlapping null and alternative distributions with a liberal alpha threshold and shaded Type I and Type II error regions. — Liberal threshold (α = 0.10) showing the tradeoff between Type I and Type II errors. Moving the critical value changes the balance between error types.

Overlapping null and alternative distributions with a standard alpha threshold and shaded Type I and Type II error regions. — Standard threshold (α = 0.05) showing the tradeoff between Type I and Type II errors. Moving the critical value changes the balance between error types.

Overlapping null and alternative distributions with a stringent alpha threshold and shaded Type I and Type II error regions. — Stringent threshold (α = 0.01) showing the tradeoff between Type I and Type II errors. Moving the critical value changes the balance between error types.

Overlapping null and alternative distributions with a very stringent alpha threshold and shaded Type I and Type II error regions. — Very stringent threshold (α = 0.001) showing the tradeoff between Type I and Type II errors. Moving the critical value changes the balance between error types.

Managing the TradeoffLink Copied

Given this tradeoff, how should you balance Type I and Type II errors? The answer depends on the relative costs of each error type in your specific context.

Framework for Choosing α:

Consider what happens if you make each type of error:

If Type I Error is...	And Type II Error is...	Then...
Very costly	Less costly	Use lower α (0.01 or lower)
Less costly	Very costly	Use higher α (0.10) and ensure adequate power
Equally costly	Equally costly	Use conventional α (0.05)

Examples of Context-Dependent Decisions:

Criminal trial: Type I error (convicting innocent) is considered much worse than Type II error (acquitting guilty). This is why "beyond reasonable doubt" sets a very high bar, effectively using a very low α.
Medical screening: For a deadly but treatable disease, Type II error (missing cases) may be worse than Type I error (false alarms that lead to follow-up testing). A higher α might be appropriate.
Drug approval: Both errors are costly: approving ineffective drugs (Type I) wastes resources and exposes patients to side effects, while rejecting effective drugs (Type II) denies patients beneficial treatments. The FDA's approach is to use stringent α but also require adequate sample sizes for power.
Particle physics: Claiming a new particle discovery based on noise would be extremely embarrassing and wasteful. The 5-sigma standard (α ≈ 3 × 10⁻⁷) reflects the high cost of Type I errors in this field.