Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Sample Size, Minimum Detectable Effect, and Power › Sample Size Formulas for Common Tests

Michael BrenndoerferJanuary 7, 20264 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 19Sample Size, Minimum Detectable Effect, and Power › Sample Size Formulas for Common Tests

Reading section 4 of 12 · Sample Size Formulas for Common Tests · 4 minSection 4 of 12 · 4 min

Section 4 of 12 · 4 min read

Sample Size, Minimum Detectable Effect, and Power › Sample Size Formulas for Common TestsOne section · Sections in this chapter4 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Different tests require slightly different formulas. Here are the most common ones:

Two-Sample t-test (Equal Variances)Link Copied

For comparing two independent groups with $n$ observations each:

n = 2 \left(\frac{z_{\alpha/2} + z_{1-\beta}}{d}\right)^2

The factor of 2 accounts for the increased variance when comparing two groups.

In[4]:

Code

def sample_size_two_sample(effect_size, sigma=1, alpha=0.05, power=0.80):
    """
    Calculate required sample size per group for a two-sample t-test.

    Parameters:
    -----------
    effect_size : float
        The minimum detectable difference between groups
    sigma : float
        Common standard deviation (assumed equal in both groups)
    alpha : float
        Significance level (two-tailed)
    power : float
        Desired power

    Returns:
    --------
    int : Required sample size per group (rounded up)
    """
    d = effect_size / sigma
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)

    # Factor of 2 for two-sample test
    n = 2 * ((z_alpha + z_beta) / d) ** 2

    return int(np.ceil(n))


# Example: Two-sample test for medium effect
n_per_group = sample_size_two_sample(
    effect_size=0.5, sigma=1, alpha=0.05, power=0.80
)
print("Two-sample t-test for d = 0.5:")
print(f"  Required n per group = {n_per_group}")
print(f"  Total N = {2 * n_per_group}")

Out[4]:

Console

Two-sample t-test for d = 0.5:
  Required n per group = 63
  Total N = 126

Paired t-testLink Copied

For paired observations (before/after, matched pairs):

n = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{d}\right)^2

where $d = \delta / \sigma_d$ and $\sigma_d$ is the standard deviation of the differences.

The paired design is typically more efficient because it removes between-subject variability.

In[5]:

Code

def sample_size_paired(effect_size, sigma_diff, alpha=0.05, power=0.80):
    """
    Calculate required number of pairs for a paired t-test.

    Parameters:
    -----------
    effect_size : float
        The minimum detectable mean difference
    sigma_diff : float
        Standard deviation of the paired differences
    alpha : float
        Significance level (two-tailed)
    power : float
        Desired power

    Returns:
    --------
    int : Required number of pairs (rounded up)
    """
    d = effect_size / sigma_diff
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)

    n = ((z_alpha + z_beta) / d) ** 2

    return int(np.ceil(n))


# Example: Paired test - before/after study
# Expect 5-point improvement, SD of differences = 10
n_pairs = sample_size_paired(
    effect_size=5, sigma_diff=10, alpha=0.05, power=0.80
)
print("Paired t-test (effect=5, SD_diff=10):")
print(f"  Required n pairs = {n_pairs}")

Out[5]:

Console

Paired t-test (effect=5, SD_diff=10):
  Required n pairs = 32

Test for Two ProportionsLink Copied

For comparing two proportions $p_1$ and $p_2$ :

n = \left(\frac{z_{\alpha/2}\sqrt{2\bar{p}(1-\bar{p})} + z_{1-\beta}\sqrt{p_1(1-p_1) + p_2(1-p_2)}}{p_1 - p_2}\right)^2

where $\bar{p} = (p_1 + p_2)/2$ .

In[6]:

Code

def sample_size_proportions(p1, p2, alpha=0.05, power=0.80):
    """
    Calculate required sample size per group for comparing two proportions.

    Parameters:
    -----------
    p1 : float
        Expected proportion in group 1 (e.g., control)
    p2 : float
        Expected proportion in group 2 (e.g., treatment)
    alpha : float
        Significance level (two-tailed)
    power : float
        Desired power

    Returns:
    --------
    int : Required sample size per group (rounded up)
    """
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)

    # Pooled proportion
    p_bar = (p1 + p2) / 2

    # Formula components
    numerator = z_alpha * np.sqrt(2 * p_bar * (1 - p_bar)) + z_beta * np.sqrt(
        p1 * (1 - p1) + p2 * (1 - p2)
    )
    denominator = abs(p1 - p2)

    n = (numerator / denominator) ** 2

    return int(np.ceil(n))


# Example: A/B test - detect 5% improvement from 10% to 15% conversion
n_per_group = sample_size_proportions(p1=0.10, p2=0.15, alpha=0.05, power=0.80)
print("Two-proportion test (10% vs 15%):")
print(f"  Required n per group = {n_per_group}")
print(f"  Total N = {2 * n_per_group}")

Out[6]:

Console

Two-proportion test (10% vs 15%):
  Required n per group = 686
  Total N = 1372

Summary TableLink Copied

Test Type	Sample Size Formula	Notes
One-sample z/t	$n = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{d}\right)^2$	d = δ/σ
Two-sample t	$n = 2\left(\frac{z_{\alpha/2} + z_{1-\beta}}{d}\right)^2$	n per group
Paired t	$n = \left(\frac{z_{\alpha/2} + z_{1-\beta}}{d}\right)^2$	n pairs, d = δ/σ_d
Two proportions	Complex formula above	n per group