Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Type I and Type II Errors › Type II Errors: The Missed Discovery

Michael BrenndoerferJanuary 6, 20269 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 18Type I and Type II Errors › Type II Errors: The Missed Discovery

Reading section 4 of 10 · Type II Errors: The Missed Discovery · 9 minSection 4 of 10 · 9 min

Section 4 of 10 · 9 min read

Type I and Type II Errors › Type II Errors: The Missed DiscoveryOne section · Sections in this chapter4 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A Type II error occurs when you fail to reject a false null hypothesis. In plain language: a real effect exists, but your test fails to detect it. You've missed a discovery.

The Probability of Type II Error: βLink Copied

The probability of a Type II error is denoted by the Greek letter beta (β):

\beta = P(\text{Fail to reject } H_0 \mid H_0 \text{ is false})

This is also a conditional probability: the probability of not rejecting the null hypothesis, given that it is false.

Computing β: The MathematicsLink Copied

Unlike α, which you simply choose, β must be calculated based on several factors. The calculation requires specifying an alternative hypothesis: you need to know what the true state of the world is to compute the probability of missing it.

Let's work through the mathematics for a one-sample z-test. Suppose:

Null hypothesis: $H_0: \mu = \mu_0$
True population mean: $\mu = \mu_1$ (where $\mu_1 \neq \mu_0$ )
Known population standard deviation: $\sigma$
Sample size: $n$
Significance level: $\alpha$ (two-tailed test)

Step 1: Find the critical values under $H_0$

Under the null hypothesis, the test statistic $Z = \frac{\bar{X} - \mu_0}{\sigma / \sqrt{n}}$ follows a standard normal distribution. The critical values for a two-tailed test at significance level α are:

z_{\text{crit}} = \pm z_{\alpha/2}

For α = 0.05: $z_{\text{crit}} = \pm 1.96$

Step 2: Convert critical z-values to critical sample means

We reject $H_0$ if $\bar{X}$ falls outside the interval:

\bar{X}_{\text{lower}} = \mu_0 - z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}

\bar{X}_{\text{upper}} = \mu_0 + z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}

Step 3: Calculate β under the alternative

A Type II error occurs when $\bar{X}$ falls in the "fail to reject" region even though the true mean is $\mu_1$ . Under the alternative:

\bar{X} \sim N\left(\mu_1, \frac{\sigma^2}{n}\right)

The probability of not rejecting $H_0$ is:

\beta = P\left(\bar{X}_{\text{lower}} < \bar{X} < \bar{X}_{\text{upper}} \mid \mu = \mu_1\right)

Standardizing using the true mean $\mu_1$ :

\beta = \Phi\left(\frac{\bar{X}_{\text{upper}} - \mu_1}{\sigma/\sqrt{n}}\right) - \Phi\left(\frac{\bar{X}_{\text{lower}} - \mu_1}{\sigma/\sqrt{n}}\right)

where $\Phi$ is the standard normal CDF.

Worked Example: Calculating βLink Copied

Let's calculate β for a concrete scenario.

Scenario: A coffee company claims their beans have a mean caffeine content of 100 mg per cup ( $H_0: \mu = 100$ ). A consumer group suspects the true content is 105 mg ( $H_1: \mu = 105$ ). They plan to test 25 cups, and caffeine content is known to have σ = 15 mg.

In[4]:

Code

import numpy as np
from scipy import stats

# Parameters
mu_0 = 100  # Null hypothesis mean
mu_1 = 105  # True mean (alternative)
sigma = 15  # Population standard deviation
n = 25  # Sample size
alpha = 0.05  # Significance level

# Standard error
se = sigma / np.sqrt(n)
print(f"Standard error: σ/√n = {sigma}/√{n} = {se:.2f}")

# Critical values for the sample mean (two-tailed test)
z_crit = stats.norm.ppf(1 - alpha / 2)
x_lower = mu_0 - z_crit * se
x_upper = mu_0 + z_crit * se
print(f"\nCritical z-value: ±{z_crit:.3f}")
print(f"Fail to reject H₀ if: {x_lower:.2f} < X̄ < {x_upper:.2f}")

# Calculate β: P(fail to reject | H₁ is true)
# This is P(x_lower < X̄ < x_upper) when X̄ ~ N(mu_1, se²)
beta = stats.norm.cdf(x_upper, loc=mu_1, scale=se) - stats.norm.cdf(
    x_lower, loc=mu_1, scale=se
)
print(f"\nType II error probability: β = {beta:.4f} ({beta * 100:.1f}%)")
print(f"Power (1 - β) = {1 - beta:.4f} ({(1 - beta) * 100:.1f}%)")

Out[4]:

Console

Standard error: σ/√n = 15/√25 = 3.00

Critical z-value: ±1.960
Fail to reject H₀ if: 94.12 < X̄ < 105.88

Type II error probability: β = 0.6152 (61.5%)
Power (1 - β) = 0.3848 (38.5%)

Let's visualize what's happening:

Out[5]:

Visualization

Two overlapping normal distributions showing the relationship between null and alternative hypotheses and the Type II error region. — Visualization of Type II error. The blue distribution shows the sampling distribution under the null hypothesis (μ = 100). The orange distribution shows the sampling distribution under the true alternative (μ = 105). The vertical dashed lines mark the critical values. The orange shaded area represents β: the probability that our sample mean falls in the ''fail to reject'' region even when the true mean is 105.

What Determines β?Link Copied

The Type II error probability depends on four interrelated factors:

1. Effect size: The larger the true effect (the distance between $\mu_0$ and $\mu_1$ ), the smaller β becomes. Larger effects are easier to detect because the alternative distribution is further from the null distribution.

2. Sample size: Larger samples decrease β. More data reduces the standard error $\sigma/\sqrt{n}$ , making both distributions narrower and easier to distinguish.

3. Significance level: Lower α means higher β, all else equal. Making it harder to reject $H_0$ (requiring more extreme evidence) also makes it harder to detect true effects.

4. Population variability: Lower population variance (smaller σ) decreases β. Less noise means the signal is easier to detect.

Out[6]:

Visualization

Line plot showing Type II error probability decreasing as effect size increases. — Larger effect sizes reduce β. The conventional target of β = 0.20 (80% power) is shown as a dashed line.

Line plot showing Type II error probability decreasing as sample size increases. — Larger sample sizes reduce β. The conventional target of β = 0.20 (80% power) is shown as a dashed line.

Line plot showing Type II error probability changing with the significance level. — Lower significance levels increase β. The conventional target of β = 0.20 (80% power) is shown as a dashed line.

Line plot showing Type II error probability increasing as population standard deviation increases. — Lower population variance reduces β. The conventional target of β = 0.20 (80% power) is shown as a dashed line.

Real-World Consequences of Type II ErrorsLink Copied

Type II errors represent missed opportunities and can have serious consequences:

Medical diagnosis: A patient with early-stage cancer is told their screening test is negative. The cancer continues to grow undetected, potentially reaching a stage where treatment is less effective.

Drug development: A pharmaceutical company abandons an effective drug because their clinical trial failed to show a statistically significant benefit. Patients are deprived of a treatment that could help them.

Safety testing: An engineer fails to detect that a structural component is weaker than specifications require. The component is used in construction, potentially leading to failure under stress.

Criminal justice: A guilty person is acquitted due to insufficient evidence. While this is preferred to convicting the innocent, it still represents a failure of the justice system to hold offenders accountable.

Research: A scientist fails to detect a real relationship in their data. The discovery is delayed or never made, slowing scientific progress.