Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Sample Size, Minimum Detectable Effect, and Power › Deriving the Sample Size Formula

Michael BrenndoerferJanuary 7, 20263 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 19Sample Size, Minimum Detectable Effect, and Power › Deriving the Sample Size Formula

Reading section 3 of 12 · Deriving the Sample Size Formula · 3 minSection 3 of 12 · 3 min

Section 3 of 12 · 3 min read

Sample Size, Minimum Detectable Effect, and Power › Deriving the Sample Size FormulaOne section · Sections in this chapter3 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's derive the sample size formula for the most common case: a one-sample z-test. The same logic extends to other tests with minor modifications.

SetupLink Copied

We want to test:

$H_0: \mu = \mu_0$ (null hypothesis)
$H_1: \mu = \mu_1$ (specific alternative, where $\mu_1 = \mu_0 + \delta$ )

With:

Known population standard deviation σ
Significance level α (two-tailed)
Desired power 1 - β

The DerivationLink Copied

Step 1: Find the rejection criterion under H₀

Under the null hypothesis, the test statistic $Z = \frac{\bar{X} - \mu_0}{\sigma/\sqrt{n}}$ follows N(0, 1).

We reject $H_0$ when $|Z| > z_{\alpha/2}$ , which corresponds to:

\bar{X} > \mu_0 + z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} \quad \text{or} \quad \bar{X} < \mu_0 - z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}

Step 2: Calculate power under H₁

Under the alternative $H_1: \mu = \mu_1$ , we want:

\text{Power} = P(\text{Reject } H_0 \mid H_1 \text{ true}) = 1 - \beta

For a one-sided test (detecting $\mu_1 > \mu_0$ ), the power is:

\text{Power} = P\left(\bar{X} > \mu_0 + z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} \mid \mu = \mu_1\right)

Under $H_1$ , $\bar{X} \sim N(\mu_1, \sigma^2/n)$ . Standardizing:

\text{Power} = P\left(Z > \frac{\mu_0 + z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}} - \mu_1}{\sigma/\sqrt{n}}\right)

= P\left(Z > z_{\alpha/2} - \frac{\mu_1 - \mu_0}{\sigma/\sqrt{n}}\right)

= P\left(Z > z_{\alpha/2} - \frac{\delta \sqrt{n}}{\sigma}\right)

Step 3: Solve for n

Setting power equal to $1 - \beta$ :

1 - \beta = P\left(Z > z_{\alpha/2} - \frac{\delta \sqrt{n}}{\sigma}\right)

This means:

z_{\alpha/2} - \frac{\delta \sqrt{n}}{\sigma} = -z_{1-\beta} = -z_\beta

Note: $z_{1-\beta}$ is the z-value such that $P(Z > z_{1-\beta}) = \beta$ , so $P(Z < z_{1-\beta}) = 1 - \beta$ .

Solving for n:

\frac{\delta \sqrt{n}}{\sigma} = z_{\alpha/2} + z_{1-\beta}

\sqrt{n} = \frac{(z_{\alpha/2} + z_{1-\beta}) \sigma}{\delta}

\boxed{n = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{\delta/\sigma}\right)^2 = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{d}\right)^2}

where $d = \delta/\sigma$ is the standardized effect size (Cohen's d).

In[3]:

Code

import numpy as np
from scipy import stats


def sample_size_one_sample(effect_size, sigma=1, alpha=0.05, power=0.80):
    """
    Calculate required sample size for a one-sample z-test.

    Parameters:
    -----------
    effect_size : float
        The minimum detectable effect in original units
    sigma : float
        Population standard deviation
    alpha : float
        Significance level (two-tailed)
    power : float
        Desired power (1 - beta)

    Returns:
    --------
    int : Required sample size (rounded up)
    """
    # Standardized effect size (Cohen's d)
    d = effect_size / sigma

    # Critical values
    z_alpha = stats.norm.ppf(1 - alpha / 2)  # Two-tailed
    z_beta = stats.norm.ppf(power)  # Power quantile

    # Sample size formula
    n = ((z_alpha + z_beta) / d) ** 2

    return int(np.ceil(n))


# Example: Detect a medium effect (d = 0.5) with 80% power at α = 0.05
n = sample_size_one_sample(effect_size=0.5, sigma=1, alpha=0.05, power=0.80)
print("One-sample z-test for d = 0.5:")
print(f"  Required n = {n}")

# Verify the components
z_alpha = stats.norm.ppf(0.975)
z_beta = stats.norm.ppf(0.80)
print("\nComponents:")
print(f"  z_α/2 = {z_alpha:.4f}")
print(f"  z_(1-β) = {z_beta:.4f}")
print(f"  Sum = {z_alpha + z_beta:.4f}")
print(
    f"  n = ({z_alpha + z_beta:.4f} / 0.5)² = {((z_alpha + z_beta) / 0.5) ** 2:.1f}"
)

Out[3]:

Console

One-sample z-test for d = 0.5:
  Required n = 32

Components:
  z_α/2 = 1.9600
  z_(1-β) = 0.8416
  Sum = 2.8016
  n = (2.8016 / 0.5)² = 31.4