Sample Size, Minimum Detectable Effect, and Power › Minimum Detectable Effect (MDE)

Michael BrenndoerferJanuary 7, 20264 min read
Machine Learning from ScratchAll chapters
Part 2 of 9Hypothesis Testing
Chapter 19Sample Size, Minimum Detectable Effect, and Power › Minimum Detectable Effect (MDE)
Reading section 5 of 12 · Minimum Detectable Effect (MDE) · 4 minSection 5 of 12 · 4 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 19 of 66

Section 5 of 12 · 4 min read

Sample Size, Minimum Detectable Effect, and Power › Minimum Detectable Effect (MDE)One section · Sections in this chapter5 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Sometimes you have a fixed sample size (due to budget, time, or available participants) and need to know: What's the smallest effect this study can reliably detect?

The minimum detectable effect (MDE) answers this question. It's the effect size threshold below which your study has inadequate power.

Deriving the MDE Formula

Rearranging the sample size formula to solve for effect size:

d=zα/2+z1−βnd = \frac{z_{\alpha/2} + z_{1-\beta}}{\sqrt{n}}

For a two-sample test:

d=zα/2+z1−βn/2d = \frac{z_{\alpha/2} + z_{1-\beta}}{\sqrt{n/2}}
In[7]:
Code
def minimum_detectable_effect(
    n, alpha=0.05, power=0.80, test_type="one_sample"
):
    """
    Calculate the minimum detectable effect size (Cohen's d).

    Parameters:
    -----------
    n : int
        Sample size (total for one-sample, per group for two-sample)
    alpha : float
        Significance level (two-tailed)
    power : float
        Desired power
    test_type : str
        'one_sample' or 'two_sample'

    Returns:
    --------
    float : Minimum detectable Cohen's d
    """
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)

    if test_type == "two_sample":
        mde = (z_alpha + z_beta) / np.sqrt(n / 2)
    else:
        mde = (z_alpha + z_beta) / np.sqrt(n)

    return mde


# Example: What can we detect with n = 50?
print("Minimum Detectable Effect (Cohen's d) with 80% power, α = 0.05:\n")
for n in [20, 50, 100, 200, 500]:
    mde_one = minimum_detectable_effect(n, test_type="one_sample")
    mde_two = minimum_detectable_effect(n, test_type="two_sample")
    print(
        f"n = {n:3d}: One-sample d = {mde_one:.3f}, Two-sample d = {mde_two:.3f}"
    )
Out[7]:
Console
Minimum Detectable Effect (Cohen's d) with 80% power, α = 0.05:

n =  20: One-sample d = 0.626, Two-sample d = 0.886
n =  50: One-sample d = 0.396, Two-sample d = 0.560
n = 100: One-sample d = 0.280, Two-sample d = 0.396
n = 200: One-sample d = 0.198, Two-sample d = 0.280
n = 500: One-sample d = 0.125, Two-sample d = 0.177
Out[8]:
Visualization
Line plot showing MDE decreasing with sample size for different power levels.
Minimum detectable effect (MDE) as a function of sample size for different power levels. The curves show how larger samples allow detection of smaller effects. At 80% power, a two-sample study with n = 100 per group can detect effects of d ≈ 0.40. The dashed lines mark conventional effect size benchmarks (small = 0.2, medium = 0.5, large = 0.8).

MDE in Practice: A/B Testing

MDE is particularly important for A/B testing in tech companies. Before running an experiment, you need to know:

  1. What's the smallest improvement worth detecting?
  2. How much traffic/time do we need to detect it?
In[9]:
Code
def ab_test_mde_proportions(n_per_group, baseline_rate, alpha=0.05, power=0.80):
    """
    Calculate MDE for an A/B test on proportions (e.g., conversion rate).

    Parameters:
    -----------
    n_per_group : int
        Sample size per group
    baseline_rate : float
        Expected conversion rate in control group
    alpha : float
        Significance level
    power : float
        Desired power

    Returns:
    --------
    tuple : (absolute MDE, relative MDE as percentage lift)
    """
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)

    # Standard error under null (pooled)
    se_null = np.sqrt(2 * baseline_rate * (1 - baseline_rate) / n_per_group)

    # Standard error under alternative (approximate)
    se_alt = se_null  # Approximation for small effects

    # MDE (absolute)
    mde_abs = (z_alpha + z_beta) * se_null

    # Relative lift
    mde_relative = mde_abs / baseline_rate * 100

    return mde_abs, mde_relative


# Example: E-commerce A/B test
baseline = 0.03  # 3% conversion rate
print(f"A/B Test MDE Analysis (baseline rate = {baseline * 100}%)\n")
print(f"{'n per group':<15} {'MDE (absolute)':<18} {'MDE (% lift)':<15}")
print("-" * 48)

for n in [1000, 5000, 10000, 50000, 100000]:
    mde_abs, mde_rel = ab_test_mde_proportions(n, baseline)
    print(f"{n:<15,} {mde_abs:<18.4f} {mde_rel:<15.1f}%")
Out[9]:
Console
A/B Test MDE Analysis (baseline rate = 3.0%)

n per group     MDE (absolute)     MDE (% lift)   
------------------------------------------------
1,000           0.0214             71.2           %
5,000           0.0096             31.9           %
10,000          0.0068             22.5           %
50,000          0.0030             10.1           %
100,000         0.0021             7.1            %

This analysis reveals an important truth: detecting small improvements in conversion rates requires very large samples. A 10% relative lift on a 3% baseline means detecting a 0.3 percentage point improvement, which requires roughly 35,000 users per group.