Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Effect Sizes and Statistical Significance › Why Effect Sizes Matter

Michael BrenndoerferJanuary 8, 20261 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 20Effect Sizes and Statistical Significance › Why Effect Sizes Matter

Reading section 2 of 11 · Why Effect Sizes Matter · 1 minSection 2 of 11 · 1 min

Section 2 of 11 · 1 min read

Effect Sizes and Statistical Significance › Why Effect Sizes MatterOne section · Sections in this chapter2 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Statistical significance has a fundamental limitation: it depends heavily on sample size. With a large enough sample, you can detect effects so tiny that they have no practical importance. With a small sample, you might miss meaningful effects.

In[2]:

Code

import numpy as np
from scipy import stats

np.random.seed(42)

# A very small true effect
true_effect = 0.1  # Cohen's d = 0.1

print("Testing the same tiny effect (d = 0.1) with different sample sizes:\n")
print(
    f"{'n per group':<15} {'t-statistic':<15} {'p-value':<15} {'Significant?':<15}"
)
print("-" * 60)

for n in [30, 100, 500, 2000, 10000]:
    # Generate data with the true effect
    group1 = np.random.normal(0, 1, n)
    group2 = np.random.normal(true_effect, 1, n)

    t_stat, p_val = stats.ttest_ind(group2, group1)
    sig = "Yes" if p_val < 0.05 else "No"

    print(f"{n:<15} {t_stat:<15.2f} {p_val:<15.4f} {sig:<15}")

print("\nThe effect size remains d = 0.1 throughout!")
print("Only the p-value changes with sample size.")

Out[2]:

Console

Testing the same tiny effect (d = 0.1) with different sample sizes:

n per group     t-statistic     p-value         Significant?   
------------------------------------------------------------
30              0.71            0.4828          No             
100             1.74            0.0837          No             
500             4.05            0.0001          Yes            
2000            1.92            0.0550          No             
10000           6.92            0.0000          Yes            

The effect size remains d = 0.1 throughout!
Only the p-value changes with sample size.

This example illustrates a fundamental truth: p-values measure evidence against the null hypothesis, not the magnitude of the effect. Effect sizes fill this gap.