Sample Size, Minimum Detectable Effect, and Power › Practical Power Analysis Tools

Michael BrenndoerferJanuary 7, 20262 min read
Machine Learning from ScratchAll chapters
Part 2 of 9Hypothesis Testing
Chapter 19Sample Size, Minimum Detectable Effect, and Power › Practical Power Analysis Tools
Reading section 9 of 12 · Practical Power Analysis Tools · 2 minSection 9 of 12 · 2 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 19 of 66

Section 9 of 12 · 2 min read

Sample Size, Minimum Detectable Effect, and Power › Practical Power Analysis ToolsOne section · Sections in this chapter9 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

While understanding the formulas is important, in practice you'll often use software tools.

Python: statsmodels

In[19]:
Code
from statsmodels.stats.power import TTestIndPower

# Two-sample t-test power analysis
analysis = TTestIndPower()

# Calculate required sample size
n = analysis.solve_power(
    effect_size=0.5,  # Cohen's d
    alpha=0.05,
    power=0.80,
    alternative="two-sided",
)
print("Two-sample t-test (d=0.5, power=80%, α=0.05):")
print(f"  Required n per group: {np.ceil(n):.0f}")

# Calculate power for given n
power = analysis.solve_power(
    effect_size=0.5,
    nobs1=64,  # n per group
    alpha=0.05,
    alternative="two-sided",
)
print("\nWith n=64 per group:")
print(f"  Achieved power: {power:.1%}")

# Calculate MDE for given n and power
mde = analysis.solve_power(
    nobs1=100, alpha=0.05, power=0.80, alternative="two-sided"
)
print("\nWith n=100 per group, 80% power:")
print(f"  Minimum detectable effect: d = {mde:.3f}")
Out[19]:
Console
Two-sample t-test (d=0.5, power=80%, α=0.05):
  Required n per group: 64

With n=64 per group:
  Achieved power: 80.1%

With n=100 per group, 80% power:
  Minimum detectable effect: d = 0.398

Power Analysis Checklist

When planning a study, work through this checklist:

  1. Define the hypothesis clearly: What exactly are you testing?

  2. Determine the appropriate test: One-sample, two-sample, paired, proportions, etc.

  3. Specify the minimum effect size of interest: Based on practical significance, not statistical convenience.

  4. Estimate population variance: From pilot data, literature, or conservative assumptions.

  5. Choose significance level: Usually α = 0.05, but consider context.

  6. Set target power: At least 80%, preferably 90% for confirmatory research.

  7. Calculate required sample size: Using appropriate formulas or software.

  8. Conduct sensitivity analysis: What if assumptions are wrong?

  9. Document all assumptions: For transparency and reproducibility.

  10. Assess feasibility: Is the required sample achievable?