Summary and Practical Guide to Hypothesis Testing › Power Analysis Quick Reference

Michael BrenndoerferJanuary 10, 20262 min read
Machine Learning from ScratchAll chapters
Part 2 of 9Hypothesis Testing
Chapter 22Summary and Practical Guide to Hypothesis Testing › Power Analysis Quick Reference
Reading section 5 of 12 · Power Analysis Quick Reference · 2 minSection 5 of 12 · 2 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 22 of 66

Section 5 of 12 · 2 min read

Summary and Practical Guide to Hypothesis Testing › Power Analysis Quick ReferenceOne section · Sections in this chapter5 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Power analysis determines the sample size needed to detect an effect of a given size with specified probability.

The Power Pentagon

Five quantities are interconnected: knowing any four determines the fifth:

  1. Sample size (n): Number of observations
  2. Effect size (d): Magnitude of the effect
  3. Significance level (α): False positive rate
  4. Power (1-β): True positive rate
  5. Variability (σ): Data spread

Sample Size Formulas

One-sample t-test:

n=(z1−α/2+z1−βd)2n = \left(\frac{z_{1-\alpha/2} + z_{1-\beta}}{d}\right)^2

Two-sample t-test (equal groups):

nper group=2(z1−α/2+z1−βd)2n_{\text{per group}} = 2\left(\frac{z_{1-\alpha/2} + z_{1-\beta}}{d}\right)^2

Two proportions:

nper group=2pˉ(1−pˉ)(z1−α/2+z1−β)2(p1−p2)2n_{\text{per group}} = \frac{2\bar{p}(1-\bar{p})(z_{1-\alpha/2} + z_{1-\beta})^2}{(p_1 - p_2)^2}

Sample Size Table (Two-Sample t-test, α = 0.05, Two-tailed)

Effect SizePower = 0.80Power = 0.90Power = 0.95
d = 0.2 (small)394 per group527 per group651 per group
d = 0.5 (medium)64 per group86 per group105 per group
d = 0.8 (large)26 per group34 per group42 per group
In[5]:
Code
# Power analysis with statsmodels
from statsmodels.stats.power import TTestIndPower

analysis = TTestIndPower()

# Calculate required sample size
n = analysis.solve_power(effect_size=0.5, power=0.8, alpha=0.05)
print(f"For d=0.5, power=0.80, α=0.05: n = {n:.0f} per group")

# Calculate power for given sample size
power = analysis.solve_power(effect_size=0.5, nobs1=50, alpha=0.05)
print(f"For d=0.5, n=50/group, α=0.05: power = {power:.2f}")

# Calculate detectable effect size
mde = analysis.solve_power(nobs1=100, power=0.8, alpha=0.05)
print(f"For n=100/group, power=0.80, α=0.05: MDE = {mde:.3f}")
Out[5]:
Console
For d=0.5, power=0.80, α=0.05: n = 64 per group
For d=0.5, n=50/group, α=0.05: power = 0.70
For n=100/group, power=0.80, α=0.05: MDE = 0.398