Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Effect Sizes and Statistical Significance › Cohen's d: The Standard Effect Size for Mean Differences

Michael BrenndoerferJanuary 8, 20265 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 20Effect Sizes and Statistical Significance › Cohen's d: The Standard Effect Size for Mean Differences

Reading section 3 of 11 · Cohen's d: The Standard Effect Size for Mean Differences · 5 minSection 3 of 11 · 5 min

Section 3 of 11 · 5 min read

Effect Sizes and Statistical Significance › Cohen's d: The Standard Effect Size for Mean DifferencesOne section · Sections in this chapter3 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The most widely used effect size for comparing two group means is Cohen's d, which expresses the difference between means in standard deviation units.

The FormulaLink Copied

d = \frac{\bar{x}_1 - \bar{x}_2}{s_p}

where:

$\bar{x}_1 - \bar{x}_2$ is the difference between group means
$s_p$ is the pooled standard deviation:

s_p = \sqrt{\frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}}

Why Standardize?Link Copied

Raw mean differences are meaningful when you understand the scale. "The treatment improved test scores by 10 points" is immediately interpretable if you know what 10 points means on that test.

But raw differences can't be compared across studies using different measures. Is a 10-point improvement on one test comparable to a 5-point improvement on another? Without knowing the variability of each test, it's impossible to say.

Standardization solves this problem. A Cohen's d of 0.5 always means "half a standard deviation," regardless of whether the original measure was test scores, reaction times, or blood pressure. This makes effect sizes comparable across completely different domains.

Cohen's BenchmarksLink Copied

Jacob Cohen proposed rough benchmarks for interpreting d:

Effect Size	Cohen's d	Interpretation
Small	0.2	Subtle, may require large samples to detect
Medium	0.5	Noticeable, often practically meaningful
Large	0.8	Substantial, usually obvious

Out[3]:

Visualization

Two nearly overlapping normal distributions separated by a small Cohen's d of 0.2. — Small effect (d = 0.2) showing mostly overlapping distributions with 92% overlap.

Two normal distributions with moderate separation, representing a Cohen's d of 0.5. — Medium effect (d = 0.5) showing moderate separation with 80% overlap.

Two clearly separated normal distributions, representing a Cohen's d of 0.8. — Large effect (d = 0.8) showing clear separation with 69% overlap.

Practical InterpretationLink Copied

Cohen's benchmarks are useful starting points, but context matters enormously. Consider:

Medical interventions: A d = 0.2 effect on mortality might save thousands of lives and justify widespread adoption.
Educational interventions: A d = 0.8 effect on test scores might not matter if the intervention costs $10,000 per student.
Business decisions: A d = 0.1 effect on conversion rates might be worth millions in a large-scale A/B test.

Always interpret effect sizes in the context of:

The practical significance of the outcome
The cost of achieving the effect
Comparison to other interventions in the field

Computing Cohen's dLink Copied

In[4]:

Code

import numpy as np
from scipy import stats


def cohens_d(group1, group2):
    """
    Calculate Cohen's d for two independent groups.

    Parameters:
    -----------
    group1 : array-like
        First group data
    group2 : array-like
        Second group data

    Returns:
    --------
    float : Cohen's d effect size
    """
    n1, n2 = len(group1), len(group2)
    var1 = np.var(group1, ddof=1)
    var2 = np.var(group2, ddof=1)

    # Pooled standard deviation
    pooled_std = np.sqrt(((n1 - 1) * var1 + (n2 - 1) * var2) / (n1 + n2 - 2))

    # Cohen's d
    d = (np.mean(group1) - np.mean(group2)) / pooled_std

    return d


# Example: Teaching method comparison
np.random.seed(42)
traditional = np.random.normal(75, 10, 30)  # Traditional method
new_method = np.random.normal(82, 10, 30)  # New method

# Calculate effect size and t-test
d = cohens_d(new_method, traditional)
t_stat, p_value = stats.ttest_ind(new_method, traditional)

print("Teaching Method Comparison")
print("=" * 40)
print(
    f"Traditional: mean = {np.mean(traditional):.1f}, SD = {np.std(traditional, ddof=1):.1f}"
)
print(
    f"New Method:  mean = {np.mean(new_method):.1f}, SD = {np.std(new_method, ddof=1):.1f}"
)
print(
    f"\nRaw difference: {np.mean(new_method) - np.mean(traditional):.1f} points"
)
print(f"Cohen's d: {d:.2f}")
print(f"t-statistic: {t_stat:.2f}")
print(f"p-value: {p_value:.4f}")
print(f"\nInterpretation: {abs(d):.2f} SD improvement (medium-large effect)")

Out[4]:

Console

Teaching Method Comparison
========================================
Traditional: mean = 73.1, SD = 9.0
New Method:  mean = 80.8, SD = 9.3

Raw difference: 7.7 points
Cohen's d: 0.84
t-statistic: 3.24
p-value: 0.0020

Interpretation: 0.84 SD improvement (medium-large effect)