Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Type I and Type II Errors › Statistical Power: 1 - β

Michael BrenndoerferJanuary 6, 20265 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 18Type I and Type II Errors › Statistical Power: 1 - β

Reading section 5 of 10 · Statistical Power: 1 - β · 5 minSection 5 of 10 · 5 min

Section 5 of 10 · 5 min read

Type I and Type II Errors › Statistical Power: 1 - βOne section · Sections in this chapter5 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Statistical power is defined as $1 - \beta$ : the probability of correctly rejecting a false null hypothesis. If β is the probability of a Type II error (missing a real effect), then power is the probability of detecting that real effect.

\text{Power} = 1 - \beta = P(\text{Reject } H_0 \mid H_0 \text{ is false})

Why Power MattersLink Copied

Power tells you how sensitive your study is. If an effect exists, power gives the probability that your study will find it:

80% power: 80% chance of detecting a true effect, 20% chance of missing it
50% power: Essentially a coin flip: you're as likely to miss the effect as to find it
20% power: You'll miss the effect 80% of the time: your study is almost useless

The conventional target is 80% power (β = 0.20). This means accepting a 1-in-5 chance of missing a real effect, which is considered an acceptable tradeoff in most research contexts. Some fields, particularly confirmatory research or high-stakes decisions, target 90% power.

Power CurvesLink Copied

A power curve shows how power varies with effect size for a given sample size and significance level. These curves are essential for understanding what effects your study can detect.

Out[7]:

Visualization

Line plot showing power curves that increase with effect size, with different curves for different sample sizes. — Power curves showing the probability of detecting an effect as a function of effect size (Cohen's d) for different sample sizes. The dashed horizontal line marks 80% power, the conventional target. Larger samples allow detection of smaller effects. With n = 10, you need a large effect (d ≈ 1.0) to reach 80% power. With n = 200, even a small effect (d ≈ 0.2) can be detected with high probability.

The Problem of Underpowered StudiesLink Copied

Underpowered studies are one of the most serious problems in research. When a study lacks sufficient power:

True effects are missed: The study is likely to produce a result that is not statistically significant even when a real effect exists.
Significant results are exaggerated: If an underpowered study does find significance, the effect size estimate is likely to be inflated (the "winner's curse").
Non-significant results are misinterpreted: Researchers may incorrectly conclude that "there is no effect" when the study simply lacked the power to detect it.
Resources are wasted: Studies that cannot answer the research question consume participant effort as well as time and money.
Publication bias is amplified: Only the "lucky" underpowered studies that happen to achieve significance get published, leading to a distorted literature.

In[8]:

Code

import numpy as np
from scipy import stats

np.random.seed(42)

# Simulate an underpowered study
# True effect: d = 0.3 (small effect)
# Sample size: n = 20 per group (typical for many studies)

n_simulations = 10000
n_per_group = 20
true_effect = 0.3  # Cohen's d
alpha = 0.05

significant_count = 0
significant_effects = []

for _ in range(n_simulations):
    # Generate data from two populations with a true difference
    group1 = np.random.normal(0, 1, n_per_group)
    group2 = np.random.normal(true_effect, 1, n_per_group)

    # Conduct t-test
    t_stat, p_value = stats.ttest_ind(group2, group1)

    if p_value < alpha:
        significant_count += 1
        # Calculate observed effect size
        pooled_std = np.sqrt(
            (np.var(group1, ddof=1) + np.var(group2, ddof=1)) / 2
        )
        observed_d = (np.mean(group2) - np.mean(group1)) / pooled_std
        significant_effects.append(observed_d)

power = significant_count / n_simulations
print(f"True effect size: d = {true_effect}")
print(f"Sample size: n = {n_per_group} per group")
print(f"Simulated power: {power:.1%}")
print(f"\nOf {n_simulations:,} studies:")
print(f"  - {significant_count:,} achieved p < 0.05")
print(f"  - {n_simulations - significant_count:,} failed to detect the effect")

if significant_effects:
    print("\nAmong significant results:")
    print(f"  - Mean observed effect: d = {np.mean(significant_effects):.3f}")
    print(
        f"  - This is {np.mean(significant_effects) / true_effect:.1f}x the true effect!"
    )

Out[8]:

Console

True effect size: d = 0.3
Sample size: n = 20 per group
Simulated power: 15.6%

Of 10,000 studies:
  - 1,559 achieved p < 0.05
  - 8,441 failed to detect the effect

Among significant results:
  - Mean observed effect: d = 0.793
  - This is 2.6x the true effect!

This simulation demonstrates the winner's curse: when underpowered studies do achieve significance, they systematically overestimate the true effect size. This happens because only the "lucky" samples, those with upward random fluctuations, cross the significance threshold.