Sample Size, Minimum Detectable Effect, and Power › The Power Analysis Framework
Machine Learning from ScratchAll chapters Sample Size, Minimum Detectable Effect, and Power › The Power Analysis FrameworkOne section · Sections in this chapter2 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Power analysis connects five interrelated quantities. Given any four, you can solve for the fifth:
The Five Quantities
1. Sample Size (n): The number of observations in your study. Larger samples provide more information and reduce uncertainty.
2. Effect Size (δ): The magnitude of the true effect you're trying to detect. This can be expressed in raw units (e.g., "10 mmHg reduction in blood pressure") or standardized units (e.g., Cohen's d = 0.5).
3. Population Variance (σ²): The natural variability in your measurements. More variance means more noise, making signals harder to detect.
4. Significance Level (α): The probability of Type I error you're willing to accept. Lower α means more stringent requirements for rejecting the null hypothesis.
5. Power (1 - β): The probability of correctly detecting a true effect. Higher power means greater sensitivity to real effects.
The Relationships
These quantities are connected by fundamental relationships that govern all hypothesis tests:
- Larger sample sizes increase power because more data reduces sampling variability
- Larger effect sizes increase power because bigger signals are easier to detect
- Higher variance decreases power because more noise obscures the signal
- Lower α decreases power because stricter criteria make rejection harder
- Targeting higher power requires larger samples to achieve greater sensitivity