Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Confidence Intervals and Test Assumptions › Confidence Intervals: A Deeper Look

Michael BrenndoerferUpdated January 12, 20265 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 13Confidence Intervals and Test Assumptions › Confidence Intervals: A Deeper Look

Reading section 2 of 9 · Confidence Intervals: A Deeper Look · 5 minSection 2 of 9 · 5 min

Section 2 of 9 · 5 min read

Confidence Intervals and Test Assumptions › Confidence Intervals: A Deeper LookOne section · Sections in this chapter2 / 9 ↓

Sections in this chapter9

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

You've probably seen confidence intervals before: "The average treatment effect was 5.2 with a 95% confidence interval of [3.1, 7.3]." But what does this mean, and how does it connect to hypothesis testing?

What a Confidence Interval Really MeansLink Copied

A 95% confidence interval does NOT mean there's a 95% probability that the true parameter lies within the interval. The true parameter is fixed (though unknown). It's either inside the interval or it isn't. There's no probability involved once you've computed the interval.

Confidence Interval Interpretation

A 95% confidence interval means: if we repeated our sampling procedure many times and computed a confidence interval each time, 95% of those intervals would contain the true parameter value.

This is a subtle but important distinction. The confidence refers to the procedure, not to any particular interval. Over many repetitions, 95% of the intervals produced by this method will capture the true value.

Out[2]:

Visualization

Multiple horizontal confidence intervals with most containing the true population mean marked by a vertical line. — Simulation showing what '95% confidence' actually means. Each horizontal line represents a 95% confidence interval from a different random sample of the same population (true mean = 50). About 95% of intervals (blue) contain the true mean, while about 5% (red) miss it. The randomness is in which samples we happen to draw, not in the parameter itself.

Constructing a Confidence Interval Step by StepLink Copied

Let's build a confidence interval from first principles. Suppose you have a sample of n observations and want a $(1-\alpha) \times 100\%$ confidence interval for the population mean.

Step 1: Calculate the sample mean

\bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i

Step 2: Calculate the standard error

The standard error measures how much the sample mean varies from sample to sample. If you know the population standard deviation $\sigma$ :

SE = \frac{\sigma}{\sqrt{n}}

If you don't know $\sigma$ (the usual case), estimate it with the sample standard deviation:

SE = \frac{s}{\sqrt{n}} \quad \text{where} \quad s = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2}

Step 3: Find the critical value

For a 95% confidence interval ( $\alpha = 0.05$ ), you need the value that cuts off 2.5% in each tail of the distribution.

If you know $\sigma$ , use the z critical value: $z_{1-\alpha/2} = z_{0.975} = 1.96$

If you're estimating $\sigma$ with $s$ , use the t critical value with $n-1$ degrees of freedom: $t_{1-\alpha/2, n-1}$

Step 4: Construct the interval

\text{CI} = \bar{x} \pm t_{1-\alpha/2, n-1} \times SE = \left[\bar{x} - t_{1-\alpha/2, n-1} \cdot \frac{s}{\sqrt{n}}, \quad \bar{x} + t_{1-\alpha/2, n-1} \cdot \frac{s}{\sqrt{n}}\right]

In[3]:

Code

import numpy as np
from scipy import stats

# Sample data: reaction times in milliseconds
reaction_times = np.array(
    [
        245,
        268,
        252,
        241,
        259,
        263,
        248,
        271,
        255,
        249,
        262,
        247,
        258,
        244,
        266,
        253,
        260,
        242,
        257,
        251,
    ]
)

# Step 1: Sample mean
n = len(reaction_times)
sample_mean = np.mean(reaction_times)

# Step 2: Standard error
sample_std = np.std(reaction_times, ddof=1)
standard_error = sample_std / np.sqrt(n)

# Step 3: Critical value for 95% CI
alpha = 0.05
t_critical = stats.t.ppf(1 - alpha / 2, df=n - 1)

# Step 4: Construct the interval
margin_of_error = t_critical * standard_error
ci_lower = sample_mean - margin_of_error
ci_upper = sample_mean + margin_of_error

print(f"Sample size: n = {n}")
print(f"Sample mean: x̄ = {sample_mean:.2f} ms")
print(f"Sample std: s = {sample_std:.2f} ms")
print(f"Standard error: SE = {standard_error:.2f} ms")
print(f"t critical value (df={n - 1}): t = {t_critical:.3f}")
print(f"Margin of error: {margin_of_error:.2f} ms")
print(f"\n95% Confidence Interval: [{ci_lower:.2f}, {ci_upper:.2f}] ms")

Out[3]:

Console

Sample size: n = 20
Sample mean: x̄ = 254.55 ms
Sample std: s = 8.80 ms
Standard error: SE = 1.97 ms
t critical value (df=19): t = 2.093
Margin of error: 4.12 ms

95% Confidence Interval: [250.43, 258.67] ms