Summary and Practical Guide to Hypothesis Testing › Common Mistakes and How to Avoid Them
Machine Learning from ScratchAll chapters Summary and Practical Guide to Hypothesis Testing › Common Mistakes and How to Avoid ThemOne section · Sections in this chapter7 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Mistake 1: P-Hacking
Problem: Running multiple analyses and only reporting significant results.
Solution: Pre-register your analysis plan. Report all tests conducted, not just significant ones. Use appropriate multiple comparison corrections.
Mistake 2: Confusing Statistical and Practical Significance
Problem: Treating p < 0.05 as proof that an effect matters.
Solution: Always report effect sizes. Ask "Is this effect large enough to be meaningful?" even when statistically significant.
Mistake 3: Misinterpreting Non-Significant Results
Problem: Concluding "no effect exists" when p ≥ 0.05.
Solution: Consider statistical power. Report confidence intervals to show the range of plausible effects. Distinguish "evidence of absence" from "absence of evidence."
Mistake 4: Violating Assumptions
Problem: Using parametric tests when assumptions are violated.
Solution: Check assumptions before testing. Use alternatives that relax those assumptions (Welch's t-test, non-parametric tests) when they fail.
Mistake 5: Ignoring Multiple Comparisons
Problem: Running many tests without correction, inflating false positive rate.
Solution: Plan your analyses in advance. Apply appropriate corrections. Report the number of tests conducted.
