Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

P-values and Hypothesis Test Setup › The Test Statistic: Standardizing Evidence

Michael BrenndoerferUpdated January 12, 20265 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 12P-values and Hypothesis Test Setup › The Test Statistic: Standardizing Evidence

Reading section 5 of 9 · The Test Statistic: Standardizing Evidence · 5 minSection 5 of 9 · 5 min

Section 5 of 9 · 5 min read

P-values and Hypothesis Test Setup › The Test Statistic: Standardizing EvidenceOne section · Sections in this chapter5 / 9 ↓

Sections in this chapter9

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Raw differences are meaningless without context. If I tell you my sample mean is 5 units above the hypothesized value, you can't assess the evidence without knowing how much variability to expect.

The test statistic solves this by standardizing the deviation. It answers: "How many standard errors away from the null hypothesis is my observation?"

The General FormulaLink Copied

For most tests involving means, the test statistic takes this form:

\text{Test Statistic} = \frac{\text{Observed value} - \text{Expected value under } H_0}{\text{Standard Error of the estimate}}

Let's break this down:

Numerator: How far is my observation from what the null predicts?

If $H_0$ claims the population mean is 100 and I observe a sample mean of 105, the numerator is $105 - 100 = 5$ .

Denominator: How much random variation should I expect?

The standard error measures the typical sampling variation. For a sample mean, it's $\sigma / \sqrt{n}$ (if we know the population standard deviation) or $s / \sqrt{n}$ (if we estimate it from the sample).

The ratio: How surprising is my observation?

A test statistic of 0.5 means my observation is half a standard error from the null. That's not surprising. A test statistic of 3 means my observation is 3 standard errors from the null. That's quite unusual.

Example: Step-by-Step CalculationLink Copied

Let's work through a complete example. Suppose you want to test whether the average weight of packages from a production line differs from the target of 500g. You sample 25 packages and measure their weights.

Step 1: State the hypotheses

$H_0: \mu = 500$ (the machine is calibrated correctly)
$H_1: \mu \neq 500$ (the machine is miscalibrated)

Step 2: Collect data and compute summary statistics

In[6]:

Code

import numpy as np

# Sample data: weights of 25 packages (in grams)
weights = np.array(
    [
        502,
        498,
        505,
        501,
        497,
        503,
        499,
        506,
        500,
        504,
        501,
        498,
        502,
        505,
        499,
        503,
        500,
        497,
        504,
        501,
        506,
        499,
        502,
        498,
        503,
    ]
)

# Summary statistics
n = len(weights)
sample_mean = np.mean(weights)
sample_std = np.std(weights, ddof=1)  # ddof=1 for sample standard deviation
standard_error = sample_std / np.sqrt(n)

print(f"Sample size: n = {n}")
print(f"Sample mean: x̄ = {sample_mean:.2f}g")
print(f"Sample standard deviation: s = {sample_std:.2f}g")
print(f"Standard error: SE = {standard_error:.2f}g")

Out[6]:

Console

Sample size: n = 25
Sample mean: x̄ = 501.32g
Sample standard deviation: s = 2.78g
Standard error: SE = 0.56g

Step 3: Calculate the test statistic

Since we don't know the population standard deviation, we use the sample standard deviation and the t-statistic:

t = \frac{\bar{x} - \mu_0}{s / \sqrt{n}} = \frac{501.36 - 500}{2.80 / \sqrt{25}} = \frac{1.36}{0.56} = 2.43

In[7]:

Code

# Calculate t-statistic
hypothesized_mean = 500
t_stat = (sample_mean - hypothesized_mean) / standard_error

print(f"t-statistic: t = {t_stat:.3f}")

Out[7]:

Console

t-statistic: t = 2.374

Step 4: Find the p-value

The t-statistic follows a t-distribution with $n - 1 = 24$ degrees of freedom. For a two-sided test, we need the probability of being as extreme or more extreme than our observed t-statistic in either direction:

\text{p-value} = 2 \times P(T > |t|) = 2 \times P(T > 2.43)

In[8]:

Code

# Calculate two-sided p-value
df = n - 1
p_value = 2 * (1 - stats.t.cdf(abs(t_stat), df=df))

print(f"Degrees of freedom: df = {df}")
print(f"Two-sided p-value: p = {p_value:.4f}")

Out[8]:

Console

Degrees of freedom: df = 24
Two-sided p-value: p = 0.0259

Step 5: Make a decision

At the conventional α = 0.05 significance level, our p-value of 0.023 is less than 0.05. We reject the null hypothesis and conclude there's statistically significant evidence that the mean weight differs from 500g.

Out[9]:

Visualization

t-distribution with observed test statistic and shaded p-value regions. — Visualization of the t-test. The t-statistic of 2.43 falls in the right tail of the t-distribution. The shaded regions show the p-value: the probability of observing a t-statistic this extreme or more extreme if the null hypothesis were true.

Why Sample Size MattersLink Copied

The standard error $s / \sqrt{n}$ shrinks as sample size grows, but not linearly. It shrinks proportionally to $\sqrt{n}$ :

4× the sample size → 2× smaller standard error
100× the sample size → 10× smaller standard error

This has important implications:

Larger samples detect smaller effects: With enough data, even tiny differences become "statistically significant."
Diminishing returns: Each additional observation contributes less than the previous one. Quadrupling your sample only halves your uncertainty.
Statistical vs practical significance: A huge sample might find a "significant" effect that's too small to matter in practice.

Out[10]:

Visualization

Line plot showing how standard error decreases with increasing sample size. — The standard error decreases with sample size, but with diminishing returns. Doubling from n=10 to n=20 cuts the standard error by about 30%, but doubling from n=100 to n=200 only cuts it by about 30% again.