Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Type I and Type II Errors › Putting It All Together: A Worked Example

Michael BrenndoerferJanuary 6, 20264 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 18Type I and Type II Errors › Putting It All Together: A Worked Example

Reading section 7 of 10 · Putting It All Together: A Worked Example · 4 minSection 7 of 10 · 4 min

Section 7 of 10 · 4 min read

Type I and Type II Errors › Putting It All Together: A Worked ExampleOne section · Sections in this chapter7 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's work through a complete example that ties together all the concepts.

Scenario: A pharmaceutical company is testing whether a new drug reduces blood pressure more than the current standard treatment. They need to design a study that balances both error types appropriately.

In[10]:

Code

import numpy as np
from scipy import stats

# Study parameters
# Current treatment: mean reduction of 10 mmHg
# New drug: suspected to reduce by 13 mmHg (3 mmHg improvement)
# Standard deviation: 8 mmHg (from previous studies)

mu_0 = 10  # Effect of current treatment (null: new drug is same)
mu_1 = 13  # Expected effect of new drug
sigma = 8  # Population standard deviation
effect = mu_1 - mu_0  # Expected improvement

print("=== Study Design Analysis ===\n")
print(f"Null hypothesis: New drug effect = {mu_0} mmHg (same as current)")
print(f"Alternative: New drug effect = {mu_1} mmHg")
print(f"Expected improvement: {effect} mmHg")
print(f"Population SD: {sigma} mmHg")

# Analysis with different sample sizes
print("\n--- Power Analysis ---")
print(f"{'n':<8} {'SE':<8} {'β':<10} {'Power':<10}")
print("-" * 40)

for n in [25, 50, 100, 150, 200]:
    se = sigma / np.sqrt(n)
    z_crit = stats.norm.ppf(0.975)  # Two-tailed, α = 0.05

    # Critical values for sample mean
    x_lower = mu_0 - z_crit * se
    x_upper = mu_0 + z_crit * se

    # Beta under alternative (focusing on upper tail since we expect mu_1 > mu_0)
    beta = stats.norm.cdf(x_upper, loc=mu_1, scale=se) - stats.norm.cdf(
        x_lower, loc=mu_1, scale=se
    )
    power = 1 - beta

    print(f"{n:<8} {se:<8.2f} {beta:<10.4f} {power:<10.1%}")

Out[10]:

Console

=== Study Design Analysis ===

Null hypothesis: New drug effect = 10 mmHg (same as current)
Alternative: New drug effect = 13 mmHg
Expected improvement: 3 mmHg
Population SD: 8 mmHg

--- Power Analysis ---
n        SE       β          Power     
----------------------------------------
25       1.60     0.5338     46.6%     
50       1.13     0.2446     75.5%     
100      0.80     0.0367     96.3%     
150      0.65     0.0042     99.6%     
200      0.57     0.0004     100.0%

Out[11]:

Visualization

Line plot showing power increasing with sample size, with 80% power marked. — Power analysis for the blood pressure drug study. The plot shows how power increases with sample size. With α = 0.05 and an expected 3 mmHg improvement (σ = 8 mmHg), approximately 100 patients per group are needed to achieve 80% power. Recruiting fewer patients risks missing a genuine clinical benefit.

Decision Framework for the ExampleLink Copied

Based on this analysis, the pharmaceutical company can make an informed decision:

In[12]:

Code

# Summary of study design options

print("=== Study Design Decision Framework ===\n")

options = [
    {
        "n": 50,
        "power": 0.56,
        "cost": "Low",
        "risk": "High (44% chance of missing a real benefit)",
    },
    {
        "n": 100,
        "power": 0.80,
        "cost": "Medium",
        "risk": "Moderate (20% chance of missing a real benefit)",
    },
    {
        "n": 150,
        "power": 0.91,
        "cost": "High",
        "risk": "Low (9% chance of missing a real benefit)",
    },
]

print("Option Analysis:")
print("-" * 70)
for opt in options:
    print(f"\nn = {opt['n']} patients per group:")
    print(f"  Power: {opt['power']:.0%}")
    print(f"  Cost: {opt['cost']}")
    print(f"  Risk: {opt['risk']}")

print("\n" + "=" * 70)
print("\nRecommendation: n = 100 per group")
print("  - Achieves conventional 80% power target")
print("  - Balances cost with acceptable Type II error risk")
print("  - If budget allows, n = 150 provides additional safety margin")

Out[12]:

Console

=== Study Design Decision Framework ===

Option Analysis:
----------------------------------------------------------------------

n = 50 patients per group:
  Power: 56%
  Cost: Low
  Risk: High (44% chance of missing a real benefit)

n = 100 patients per group:
  Power: 80%
  Cost: Medium
  Risk: Moderate (20% chance of missing a real benefit)

n = 150 patients per group:
  Power: 91%
  Cost: High
  Risk: Low (9% chance of missing a real benefit)

======================================================================

Recommendation: n = 100 per group
  - Achieves conventional 80% power target
  - Balances cost with acceptable Type II error risk
  - If budget allows, n = 150 provides additional safety margin