Sample Size, Minimum Detectable Effect, and Power › The Problem of Underpowered Studies
Machine Learning from ScratchAll chapters Sample Size, Minimum Detectable Effect, and Power › The Problem of Underpowered StudiesOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Underpowered studies are one of the most pervasive problems in research. When a study has insufficient power:
Consequences of Low Power
1. High False Negative Rate: Real effects are missed because the study lacks sensitivity to detect them. This wastes resources and may lead to abandoning potentially useful treatments or interventions.
2. Winner's Curse (Effect Size Inflation): When an underpowered study does achieve significance, the estimated effect size is systematically inflated. Only the "lucky" samples with upward random fluctuations cross the significance threshold.
import numpy as np
from scipy import stats
np.random.seed(42)
# Simulate many underpowered studies
true_effect = 0.3 # Small true effect (Cohen's d)
n_per_group = 20 # Small sample (underpowered for this effect)
n_simulations = 10000
significant_effects = []
all_effects = []
for _ in range(n_simulations):
# Generate data
group1 = np.random.normal(0, 1, n_per_group)
group2 = np.random.normal(true_effect, 1, n_per_group)
# Calculate observed effect
pooled_std = np.sqrt((np.var(group1, ddof=1) + np.var(group2, ddof=1)) / 2)
observed_d = (np.mean(group2) - np.mean(group1)) / pooled_std
all_effects.append(observed_d)
# Test significance
_, p = stats.ttest_ind(group2, group1)
if p < 0.05:
significant_effects.append(observed_d)
power = len(significant_effects) / n_simulations
mean_significant = np.mean(significant_effects) if significant_effects else 0
inflation = (mean_significant / true_effect - 1) * 100
print("=== Winner's Curse Simulation ===\n")
print(f"True effect size: d = {true_effect}")
print(f"Sample size: n = {n_per_group} per group (underpowered)")
print(f"Simulated power: {power:.1%}")
print()
print(f"Among {len(significant_effects):,} significant results:")
print(f" Mean observed effect: d = {mean_significant:.3f}")
print(f" True effect: d = {true_effect:.3f}")
print(f" Inflation: {inflation:.0f}%")=== Winner's Curse Simulation === True effect size: d = 0.3 Sample size: n = 20 per group (underpowered) Simulated power: 15.6% Among 1,559 significant results: Mean observed effect: d = 0.793 True effect: d = 0.300 Inflation: 164%
- Non-Replicability: Inflated effect sizes from underpowered studies are unlikely to replicate. Follow-up studies with the same sample size will often fail to reach significance."
4. Wasted Resources: Underpowered studies waste participants' effort as well as time and money on research that cannot reliably answer the question being asked.
Avoiding Underpowered Studies
The solution is straightforward: conduct power analysis before data collection. This involves:
-
Specify the smallest effect size of practical importance: What's the minimum effect worth detecting? This is a scientific/business question, not a statistical one.
-
Choose target power: The convention is 80%, but 90% is better for confirmatory research.
-
Calculate required sample size: Use the formulas and tools covered in this section.
-
If required sample is infeasible: Either accept lower power (with documented limitations), seek more funding/participants, or consider whether the study should be conducted at all.
An underpowered study with uncertain results is often worse than no study at all, because it can mislead future research and policy decisions.