Type I and Type II Errors › The Two Ways Tests Can Fail

Michael BrenndoerferJanuary 6, 20264 min read
Machine Learning from ScratchAll chapters
Part 2 of 9Hypothesis Testing
Chapter 18Type I and Type II Errors › The Two Ways Tests Can Fail
Reading section 2 of 10 · The Two Ways Tests Can Fail · 4 minSection 2 of 10 · 4 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 18 of 66

Section 2 of 10 · 4 min read

Type I and Type II Errors › The Two Ways Tests Can FailOne section · Sections in this chapter2 / 10 ↓
Sections in this chapter10
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When we conduct a hypothesis test, we are making a binary decision: reject the null hypothesis or fail to reject it. Reality also has two states: either the null hypothesis is true, or it is false. This creates a 2×2 matrix of possible outcomes.

Out[2]:
Visualization
A 2x2 matrix showing the four possible outcomes of hypothesis testing decisions.
The four possible outcomes when making a decision based on a hypothesis test. Correct decisions occur when our conclusion matches reality. Errors occur when they do not. Type I errors (false positives) happen when we reject a true null hypothesis. Type II errors (false negatives) happen when we fail to reject a false null hypothesis.

Let's define these four outcomes precisely:

  1. True Negative: The null hypothesis is true (no effect exists), and we correctly fail to reject it. This is a correct decision.

  2. True Positive: The null hypothesis is false (an effect exists), and we correctly reject it. This is a correct decision, and its probability is called power.

  3. Type I Error (False Positive): The null hypothesis is true (no effect exists), but we incorrectly reject it. We claim to have found something that isn't there.

  4. Type II Error (False Negative): The null hypothesis is false (an effect exists), but we fail to reject it. We miss an effect that is there.