Summary and Practical Guide to Hypothesis Testing › The Complete Hypothesis Testing Workflow

Michael BrenndoerferJanuary 10, 20264 min read
Machine Learning from ScratchAll chapters
Part 2 of 9Hypothesis Testing
Chapter 22Summary and Practical Guide to Hypothesis Testing › The Complete Hypothesis Testing Workflow
Reading section 2 of 12 · The Complete Hypothesis Testing Workflow · 4 minSection 2 of 12 · 4 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 22 of 66

Section 2 of 12 · 4 min read

Summary and Practical Guide to Hypothesis Testing › The Complete Hypothesis Testing WorkflowOne section · Sections in this chapter2 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before diving into details, here's the complete workflow for conducting a hypothesis test. Each step is critical: skipping any one can invalidate your conclusions.

Out[3]:
Visualization
Flowchart showing eight sequential hypothesis-testing steps, grouped into before-data, data-collection, analysis, and communication phases.
The complete hypothesis testing workflow. Following these steps in order ensures valid and interpretable results. Note that many decisions (hypotheses, α level, sample size) should be made before data collection.

Step-by-Step Guide

Step 1: Formulate Hypotheses

  • Define H0H_0 (null hypothesis): The default assumption, typically "no effect" or "no difference"
  • Define H1H_1 (alternative hypothesis): What you're trying to demonstrate
  • Choose one-tailed or two-tailed based on your research question
  • Do this BEFORE seeing the data

Step 2: Choose Significance Level (α)

  • Standard: α = 0.05 (5% false positive rate)
  • Stringent: α = 0.01 for high-stakes decisions
  • Lenient: α = 0.10 for exploratory research
  • Consider the consequences of Type I vs Type II errors

Step 3: Determine Sample Size

  • Use power analysis to calculate required n
  • Specify: α, power (typically 0.80), minimum effect size of interest
  • Balance statistical needs against practical constraints

Step 4: Collect Data

  • Use appropriate randomization and sampling methods
  • Ensure independence of observations
  • Avoid peeking at results during collection

Step 5: Check Assumptions

  • Normality: Shapiro-Wilk test, Q-Q plots
  • Equal variances: Levene's test
  • Independence: Study design consideration
  • Choose alternatives that tolerate assumption violations

Step 6: Select and Compute Test

Step 7: Make Decision

  • If p < α: Reject H₀, conclude evidence for H₁
  • If p ≥ α: Fail to reject H₀, insufficient evidence
  • Remember: "fail to reject" ≠ "accept H₀"

Step 8: Report Results

  • Effect size (Cohen's d, η², r)
  • Confidence interval
  • Exact p-value
  • Test statistic and degrees of freedom
  • Assumption check results