Effect Sizes and Statistical Significance › Confidence Intervals for Effect Sizes
Michael BrenndoerferJanuary 8, 20263 min read
Machine Learning from ScratchAll chapters Part 2 of 9Hypothesis Testing
Effect Sizes and Statistical Significance › Confidence Intervals for Effect SizesOne section · Sections in this chapter6 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Point estimates of effect sizes have uncertainty. Confidence intervals provide a range of plausible values:
In[11]:
Codedef cohens_d_ci(group1, group2, confidence=0.95):
"""
Calculate Cohen's d with confidence interval.
Uses the non-central t-distribution approach.
"""
n1, n2 = len(group1), len(group2)
d = cohens_d(group1, group2)
# Standard error of d
se_d = np.sqrt((n1 + n2) / (n1 * n2) + d**2 / (2 * (n1 + n2)))
# t critical value
df = n1 + n2 - 2
alpha = 1 - confidence
t_crit = stats.t.ppf(1 - alpha / 2, df)
# Confidence interval
ci_lower = d - t_crit * se_d
ci_upper = d + t_crit * se_d
return d, ci_lower, ci_upper
# Example
np.random.seed(42)
group1 = np.random.normal(100, 15, 50)
group2 = np.random.normal(108, 15, 50)
d, ci_low, ci_high = cohens_d_ci(group1, group2)
print("Effect Size with 95% Confidence Interval")
print("=" * 40)
print(f"Cohen's d = {d:.3f}")
print(f"95% CI: [{ci_low:.3f}, {ci_high:.3f}]")
print(
f"\nThe true effect size is likely between {ci_low:.2f} and {ci_high:.2f}"
)Out[11]:
ConsoleEffect Size with 95% Confidence Interval ======================================== Cohen's d = -0.859 95% CI: [-1.273, -0.444] The true effect size is likely between -1.27 and -0.44
Out[12]:
Visualization