Effect Sizes and Statistical Significance › Statistical vs. Practical Significance
Michael BrenndoerferJanuary 8, 20264 min read
Machine Learning from ScratchAll chapters Part 2 of 9Hypothesis Testing
Effect Sizes and Statistical Significance › Statistical vs. Practical SignificanceOne section · Sections in this chapter7 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Sound interpretation depends on distinguishing statistical from practical significance.
Statistical Significance
- Answers: "Is the effect distinguishable from zero?"
- Depends on: Sample size, effect size, variability
- Limitation: Achievable for any non-zero effect with large enough n
Practical Significance
- Answers: "Is the effect large enough to matter?"
- Depends on: Context, costs, benefits, alternatives
- Limitation: Requires domain knowledge to interpret
Out[13]:
Visualization
A Complete Example
In[14]:
Codeimport numpy as np
from scipy import stats
np.random.seed(42)
# Scenario: Testing a new website design
# Baseline conversion rate: 5%
# We want to detect a 10% relative improvement (0.5 percentage points)
# Simulate A/B test with very large sample
n = 100000
baseline_rate = 0.05
true_improvement = 0.005 # 0.5 percentage points
control_conversions = np.random.binomial(1, baseline_rate, n)
treatment_conversions = np.random.binomial(
1, baseline_rate + true_improvement, n
)
# Calculate statistics
control_rate = control_conversions.mean()
treatment_rate = treatment_conversions.mean()
# Chi-square test
contingency = [
[control_conversions.sum(), n - control_conversions.sum()],
[treatment_conversions.sum(), n - treatment_conversions.sum()],
]
chi2, p_val, _, _ = stats.chi2_contingency(contingency)
# Effect size (Cohen's h)
h = 2 * np.arcsin(np.sqrt(treatment_rate)) - 2 * np.arcsin(
np.sqrt(control_rate)
)
# Relative lift
lift = (treatment_rate - control_rate) / control_rate * 100
print("A/B Test Results: Website Redesign")
print("=" * 50)
print(f"Sample size: {n:,} per group")
print("\nConversion Rates:")
print(f" Control: {control_rate * 100:.3f}%")
print(f" Treatment: {treatment_rate * 100:.3f}%")
print("\nStatistical Test:")
print(f" χ² = {chi2:.2f}, p = {p_val:.6f}")
print(f" Statistically significant? {'Yes' if p_val < 0.05 else 'No'}")
print("\nEffect Size:")
print(
f" Absolute difference: {(treatment_rate - control_rate) * 100:.3f} percentage points"
)
print(f" Relative lift: {lift:.1f}%")
print(f" Cohen's h: {abs(h):.4f}")
print("\n" + "=" * 50)
print("INTERPRETATION:")
print("=" * 50)
print("The result is statistically significant (p < 0.05),")
print(f"but the effect size is tiny (h = {abs(h):.3f}).")
print("\nWith 100,000 users per group, we reliably detected")
print(f"a real improvement of {lift:.1f}%, but you must ask:")
print(f"Is a {lift:.1f}% lift worth the cost of the redesign?")Out[14]:
ConsoleA/B Test Results: Website Redesign ================================================== Sample size: 100,000 per group Conversion Rates: Control: 4.871% Treatment: 5.530% Statistical Test: χ² = 43.91, p = 0.000000 Statistically significant? Yes Effect Size: Absolute difference: 0.659 percentage points Relative lift: 13.5% Cohen's h: 0.0297 ================================================== INTERPRETATION: ================================================== The result is statistically significant (p < 0.05), but the effect size is tiny (h = 0.030). With 100,000 users per group, we reliably detected a real improvement of 13.5%, but you must ask: Is a 13.5% lift worth the cost of the redesign?