P-values and Hypothesis Test Setup › The Decision Process

Michael BrenndoerferUpdated January 12, 20261 min read
Machine Learning from ScratchAll chapters
Part 2 of 9Hypothesis Testing
Chapter 12P-values and Hypothesis Test Setup › The Decision Process
Reading section 6 of 9 · The Decision Process · 1 minSection 6 of 9 · 1 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 12 of 66

Section 6 of 9 · 1 min read

P-values and Hypothesis Test Setup › The Decision ProcessOne section · Sections in this chapter6 / 9 ↓
Sections in this chapter9
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

We've calculated our test statistic and p-value. Now what? The decision follows this logic:

  1. Before seeing data: Choose a significance level α (conventionally 0.05, but context-dependent)

  2. After computing p-value:

    • If p ≤ α: Reject H0H_0. The data provide sufficient evidence against the null.
    • If p > α: Fail to reject H0H_0. The data don't provide sufficient evidence.

  3. Report results: Include the test statistic, p-value, sample size, and effect size. Never just say "significant" or "not significant."

The Critical Value Approach

An equivalent approach uses critical values instead of p-values. The critical value is the test statistic boundary separating "reject" from "fail to reject."

For a two-sided z-test at α = 0.05, the critical values are ±1.96. You reject H0H_0 if |z| > 1.96.

For a two-sided t-test, critical values depend on degrees of freedom. With df = 24 and α = 0.05, the critical values are approximately ±2.064.

Both approaches give the same answer. The p-value approach is more informative because it tells you exactly how extreme your result was.