Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Effect Sizes and Statistical Significance › Effect Sizes for Other Designs

Michael BrenndoerferJanuary 8, 20264 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 20Effect Sizes and Statistical Significance › Effect Sizes for Other Designs

Reading section 5 of 11 · Effect Sizes for Other Designs · 4 minSection 5 of 11 · 4 min

Section 5 of 11 · 4 min read

Effect Sizes and Statistical Significance › Effect Sizes for Other DesignsOne section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Paired Samples: Cohen's d_zLink Copied

For paired designs (before/after, matched pairs), standardize by the standard deviation of differences:

d_z = \frac{\bar{d}}{s_d}

where $\bar{d}$ is the mean of pairwise differences and $s_d$ is their standard deviation.

In[7]:

Code

def cohens_d_paired(before, after):
    """
    Calculate Cohen's d for paired samples (d_z).
    """
    differences = np.array(after) - np.array(before)
    return np.mean(differences) / np.std(differences, ddof=1)


# Example: Weight loss program
np.random.seed(42)
n = 25
before = np.random.normal(180, 20, n)
after = before - np.random.normal(8, 5, n)  # Average loss of ~8 lbs

d_z = cohens_d_paired(before, after)
t_stat, p_val = stats.ttest_rel(after, before)

print("Weight Loss Program Results")
print("=" * 40)
print(f"Before: {np.mean(before):.1f} ± {np.std(before, ddof=1):.1f} lbs")
print(f"After:  {np.mean(after):.1f} ± {np.std(after, ddof=1):.1f} lbs")
print(f"\nMean loss: {np.mean(before) - np.mean(after):.1f} lbs")
print(f"Cohen's d_z: {abs(d_z):.2f}")
print(f"t({n - 1}) = {abs(t_stat):.2f}, p = {p_val:.4f}")

Out[7]:

Console

Weight Loss Program Results
========================================
Before: 176.7 ± 19.1 lbs
After:  170.2 ± 18.4 lbs

Mean loss: 6.6 lbs
Cohen's d_z: 1.42
t(24) = 7.09, p = 0.0000

ANOVA: Eta-Squared and Omega-SquaredLink Copied

For ANOVA, effect sizes describe the proportion of variance explained:

Eta-squared (η²):

\eta^2 = \frac{SS_{\text{between}}}{SS_{\text{total}}}

Omega-squared (ω²), less biased:

\omega^2 = \frac{SS_{\text{between}} - df_{\text{between}} \cdot MS_{\text{within}}}{SS_{\text{total}} + MS_{\text{within}}}

Effect Size	η² / ω²	Interpretation
Small	0.01	1% of variance explained
Medium	0.06	6% of variance explained
Large	0.14	14% of variance explained

In[8]:

Code

def eta_squared(groups):
    """Calculate eta-squared from a list of groups."""
    all_data = np.concatenate(groups)
    grand_mean = np.mean(all_data)

    ss_between = sum(len(g) * (np.mean(g) - grand_mean) ** 2 for g in groups)
    ss_total = np.sum((all_data - grand_mean) ** 2)

    return ss_between / ss_total


def omega_squared(groups):
    """Calculate omega-squared (bias-corrected eta-squared)."""
    all_data = np.concatenate(groups)
    grand_mean = np.mean(all_data)
    k = len(groups)  # Number of groups
    N = len(all_data)

    ss_between = sum(len(g) * (np.mean(g) - grand_mean) ** 2 for g in groups)
    ss_within = sum(np.sum((g - np.mean(g)) ** 2) for g in groups)
    ss_total = ss_between + ss_within

    df_between = k - 1
    df_within = N - k
    ms_within = ss_within / df_within

    return (ss_between - df_between * ms_within) / (ss_total + ms_within)


# Example: Three teaching methods
np.random.seed(42)
method_a = np.random.normal(70, 10, 30)
method_b = np.random.normal(75, 10, 30)
method_c = np.random.normal(80, 10, 30)

groups = [method_a, method_b, method_c]

# ANOVA
f_stat, p_val = stats.f_oneway(*groups)
eta_sq = eta_squared(groups)
omega_sq = omega_squared(groups)

print("Three Teaching Methods ANOVA")
print("=" * 40)
print(f"Method A: {np.mean(method_a):.1f} ± {np.std(method_a, ddof=1):.1f}")
print(f"Method B: {np.mean(method_b):.1f} ± {np.std(method_b, ddof=1):.1f}")
print(f"Method C: {np.mean(method_c):.1f} ± {np.std(method_c, ddof=1):.1f}")
print(f"\nF({2}, {87}) = {f_stat:.2f}, p = {p_val:.4f}")
print(f"η² = {eta_sq:.3f} ({eta_sq * 100:.1f}% variance explained)")
print(
    f"ω² = {omega_sq:.3f} ({omega_sq * 100:.1f}% variance explained, bias-corrected)"
)

Out[8]:

Console

Three Teaching Methods ANOVA
========================================
Method A: 68.1 ± 9.0
Method B: 73.8 ± 9.3
Method C: 80.1 ± 9.9

F(2, 87) = 12.21, p = 0.0000
η² = 0.219 (21.9% variance explained)
ω² = 0.199 (19.9% variance explained, bias-corrected)

Correlation: r as an Effect SizeLink Copied

Pearson's correlation coefficient r is itself an effect size, measuring the strength of linear relationship:

Effect Size	r	Interpretation
Small	0.1	Weak relationship
Medium	0.3	Moderate relationship
Large	0.5	Strong relationship

The coefficient of determination r² gives the proportion of variance shared between variables.

In[9]:

Code

np.random.seed(42)

# Generate correlated data with different r values
n = 100

print("Correlation as Effect Size")
print("=" * 40)
print(f"{'True r':<15} {'Observed r':<15} {'r²':<15} {'Interpretation':<20}")
print("-" * 65)

for true_r, interp in [(0.1, "Small"), (0.3, "Medium"), (0.5, "Large")]:
    # Generate correlated data
    x = np.random.normal(0, 1, n)
    y = true_r * x + np.sqrt(1 - true_r**2) * np.random.normal(0, 1, n)

    r, p = stats.pearsonr(x, y)
    print(f"{true_r:<15.1f} {r:<15.3f} {r**2:<15.3f} {interp:<20}")

Out[9]:

Console

Correlation as Effect Size
========================================
True r          Observed r      r²              Interpretation      
-----------------------------------------------------------------
0.1             -0.041          0.002           Small               
0.3             0.360           0.129           Medium              
0.5             0.501           0.251           Large

Proportions: Cohen's h and Odds RatioLink Copied

For comparing proportions, several effect sizes are available:

Cohen's h (arcsine transformation):

h = 2 \arcsin(\sqrt{p_1}) - 2 \arcsin(\sqrt{p_2})

Odds Ratio:

OR = \frac{p_1 / (1 - p_1)}{p_2 / (1 - p_2)}

In[10]:

Code

def cohens_h(p1, p2):
    """Calculate Cohen's h for comparing two proportions."""
    phi1 = 2 * np.arcsin(np.sqrt(p1))
    phi2 = 2 * np.arcsin(np.sqrt(p2))
    return phi1 - phi2


def odds_ratio(p1, p2):
    """Calculate odds ratio for two proportions."""
    odds1 = p1 / (1 - p1)
    odds2 = p2 / (1 - p2)
    return odds1 / odds2


# Example: A/B test conversion rates
control_rate = 0.10
treatment_rate = 0.15

h = cohens_h(treatment_rate, control_rate)
or_ = odds_ratio(treatment_rate, control_rate)
relative_lift = (treatment_rate - control_rate) / control_rate * 100

print("A/B Test: Conversion Rate Improvement")
print("=" * 40)
print(f"Control rate: {control_rate * 100:.1f}%")
print(f"Treatment rate: {treatment_rate * 100:.1f}%")
print(
    f"\nAbsolute difference: {(treatment_rate - control_rate) * 100:.1f} percentage points"
)
print(f"Relative lift: {relative_lift:.1f}%")
print(f"Cohen's h: {h:.3f}")
print(f"Odds ratio: {or_:.2f}")
print(
    f"\nInterpretation: h = {abs(h):.2f} is a {'small' if abs(h) < 0.3 else 'medium' if abs(h) < 0.5 else 'large'} effect"
)

Out[10]:

Console

A/B Test: Conversion Rate Improvement
========================================
Control rate: 10.0%
Treatment rate: 15.0%

Absolute difference: 5.0 percentage points
Relative lift: 50.0%
Cohen's h: 0.152
Odds ratio: 1.59

Interpretation: h = 0.15 is a small effect