P-values and Hypothesis Test Setup › What a P-value Is (and Isn't)

Michael BrenndoerferUpdated January 12, 20267 min read
Machine Learning from ScratchAll chapters
Part 2 of 9Hypothesis Testing
Chapter 12P-values and Hypothesis Test Setup › What a P-value Is (and Isn't)
Reading section 3 of 9 · What a P-value Is (and Isn't) · 7 minSection 3 of 9 · 7 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 12 of 66

Section 3 of 9 · 7 min read

P-values and Hypothesis Test Setup › What a P-value Is (and Isn't)One section · Sections in this chapter3 / 9 ↓
Sections in this chapter9
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The p-value is the most misunderstood concept in statistics, but it becomes simple once you internalize what question it answers.

P-value Definition

The p-value is the probability of observing data at least as extreme as what you observed, assuming the null hypothesis is true.

Let's unpack this carefully with our coin example. You flipped 100 times, got 63 heads, and the null hypothesis says P(heads) = 0.5. The p-value asks: if the coin really were fair, how often would you see 63 or more heads (or, for a two-sided test, also 37 or fewer heads)?

Out[2]:
Visualization
Histogram showing binomial distribution of coin flips with shaded p-value region.
Distribution of heads in 100 flips of a fair coin. The vertical dashed line marks the observed result of 63 heads. The red shaded regions show outcomes as extreme or more extreme than what was observed. The total area of these regions is the two-tailed p-value.

The p-value of about 0.012 tells us that if the coin were fair, we'd see a result this extreme only about 1.2% of the time. That's pretty surprising! It doesn't prove the coin is biased, but it suggests the "fair coin" assumption doesn't fit our data very well.

Computing the P-value Step by Step

Let's work through the math explicitly. Under the null hypothesis that P(heads) = 0.5, the number of heads X follows a binomial distribution with n = 100 and p = 0.5:

P(X=k)=(100k)(0.5)k(0.5)100−k=(100k)(0.5)100P(X = k) = \binom{100}{k} (0.5)^k (0.5)^{100-k} = \binom{100}{k} (0.5)^{100}

For a two-tailed test, we want the probability of being as extreme or more extreme than 63 in either direction. Since 63 is 13 away from the expected value of 50, we count outcomes of 63 or more heads, OR 37 or fewer heads:

p-value=P(X≥63)+P(X≤37)\text{p-value} = P(X \geq 63) + P(X \leq 37)

By symmetry of the binomial distribution when p = 0.5:

p-value=2×P(X≥63)=2×∑k=63100(100k)(0.5)100\text{p-value} = 2 \times P(X \geq 63) = 2 \times \sum_{k=63}^{100} \binom{100}{k} (0.5)^{100}
In[3]:
Code
from scipy import stats

# Calculate exact p-value using binomial distribution
n = 100
p = 0.5
observed = 63

# P(X >= 63) = 1 - P(X <= 62) = 1 - CDF(62)
p_right_tail = 1 - stats.binom.cdf(observed - 1, n, p)

# Two-tailed p-value (by symmetry)
p_value = 2 * p_right_tail

print(f"P(X ≥ 63) = {p_right_tail:.6f}")
print(f"Two-tailed p-value = {p_value:.6f}")
Out[3]:
Console
P(X ≥ 63) = 0.006016
Two-tailed p-value = 0.012033

What the P-value Does NOT Mean

Understanding what the p-value isn't is as important as understanding what it is. These three misinterpretations are incredibly common, even among experienced practitioners:

Misinterpretation 1: "The p-value is the probability that the null hypothesis is true."

This is wrong. The null hypothesis is either true or false. It's a fixed (but unknown) fact about reality, not a random variable. The p-value is about the probability of the data given the hypothesis, not the probability of the hypothesis given the data.

If you want to know P(hypothesis | data), you need Bayesian methods with prior probabilities. Classical hypothesis testing doesn't give you this.

Misinterpretation 2: "1 minus the p-value is the probability the effect is real."

A p-value of 0.02 also does not mean there's a 98% chance the alternative hypothesis is true. This is the same error as above, just phrased differently.

Misinterpretation 3: "A small p-value means the effect is large or important."

This interpretation is wrong because a tiny p-value can arise from a tiny effect with a huge sample size. If you test a million users, even a 0.001% difference in conversion rates might be "statistically significant" with p < 0.001, but that doesn't make it practically meaningful.

The p-value measures evidence against the null hypothesis, not the size or importance of an effect. Effect sizes, covered in a later chapter, address that question.

Why P < 0.05 Is a Convention, Not a Law

The threshold of 0.05 for "statistical significance" comes from Ronald Fisher, who in 1925 wrote that results below this threshold were "worth taking seriously." It was never meant to be a rigid cutoff.

Yet the 0.05 threshold has calcified into a binary decision rule that distorts research. A p-value of 0.049 gets published as "significant," while 0.051 gets buried as "not significant," despite conveying nearly identical evidence. This cliff encourages p-hacking, selective reporting, and publication bias.

Better practice is to:

  • Report exact p-values, not just "p < 0.05"
  • Interpret p-values on a continuum (0.001 is much stronger evidence than 0.04)
  • Choose thresholds based on context (physics uses 5-sigma ≈ p < 0.0000003 for discoveries)
  • Consider effect sizes alongside p-values
Out[4]:
Visualization
Bar chart showing evidence strength for different p-value ranges.
P-values should be interpreted as a continuum of evidence, not as a binary significant/not-significant split. The strength of evidence increases as p-values get smaller.