Statistical Inference › Hypothesis Testing: Evaluating Claims

While estimation focuses on approximating parameter values, hypothesis testing evaluates specific claims or hypotheses about populations. This framework allows researchers to determine whether observed data provide sufficient evidence to reject a stated hypothesis, controlling error rates in a rigorous manner.

Hypothesis testing begins by formulating two competing hypotheses. The null hypothesis ($H_0$) represents a default position, skeptical claim, or status quo that we seek evidence against. The alternative hypothesis ($H_1$ or $H_a$) represents the research hypothesis or claim we wish to establish. For example, when testing whether a new teaching method improves test scores compared to the standard approach, the null hypothesis might state that the methods produce equal average scores, while the alternative claims the new method yields higher scores.

The logic of hypothesis testing follows a proof-by-contradiction structure. We assume the null hypothesis is true, then calculate how likely the observed data (or more extreme data) would be under this assumption. If the data would be highly unlikely under the null hypothesis, we have evidence to reject it in favor of the alternative. If the data are reasonably consistent with the null hypothesis, we fail to reject it, though importantly, this does not prove the null hypothesis true.

A test statistic summarizes how far the sample data deviate from what the null hypothesis predicts, measured in standard error units. For testing a population mean, the t-statistic is commonly used:

Where:

• $\bar{x}$: Sample mean
• $\mu_0$: Hypothesized population mean under $H_0$
• $s$: Sample standard deviation
• $n$: Sample size

This statistic measures how many standard errors the sample mean lies from the hypothesized value. Large absolute values of the test statistic indicate data that would be unusual if the null hypothesis were true.

The p-value quantifies the strength of evidence against the null hypothesis. Formally, the p-value is the probability of observing a test statistic at least as extreme as the one calculated from the sample data, assuming the null hypothesis is true. A small p-value indicates that the observed data would be quite unlikely under the null hypothesis.

Common misinterpretations of p-values plague statistical practice. The p-value is not the probability that the null hypothesis is true (that probability is either 0 or 1, though unknown to us). Nor is one minus the p-value the probability that the alternative hypothesis is true. The p-value also does not measure the size or importance of an effect; even trivial differences can yield tiny p-values with sufficiently large samples. Rather, the p-value simply measures how compatible the data are with the null hypothesis.

The significance level $\alpha$ (alpha) sets a threshold for decision making. Common choices include $\alpha = 0.05$ or $\alpha = 0.01$. If the p-value falls below $\alpha$, we reject the null hypothesis and declare the result "statistically significant." If the p-value exceeds $\alpha$, we fail to reject the null hypothesis.

The significance level represents the maximum Type I error rate we are willing to tolerate. It should be chosen before conducting the test based on the consequences of different types of errors in the specific context, not after seeing the data. Conventions like $\alpha = 0.05$ are merely defaults; situations with severe consequences for false positives might warrant more stringent levels like $\alpha = 0.01$, while exploratory analyses might use $\alpha = 0.10$.

Hypothesis testing involves making decisions under uncertainty, which inevitably leads to the possibility of errors. A Type I error occurs when we reject a true null hypothesis, a "false positive" that claims an effect exists when it does not. The probability of a Type I error equals the significance level $\alpha$, which we control directly by our choice of threshold.

A Type II error occurs when we fail to reject a false null hypothesis, a "false negative" that misses a real effect. The probability of a Type II error is denoted $\beta$ (beta) and depends on the true parameter value, sample size, and significance level. The power of a test, defined as $1 - \beta$, represents the probability of correctly rejecting a false null hypothesis. Higher power means the test is more likely to detect true effects when they exist.

These error types reflect a fundamental tradeoff. Making it harder to commit Type I errors (lowering $\alpha$) simultaneously makes Type II errors more likely unless we increase sample size. The appropriate balance depends on the relative costs of each error type in context. Medical screening tests for serious diseases might prioritize avoiding Type II errors (missing true cases) even at the cost of more Type I errors (false alarms), while legal systems traditionally emphasize minimizing Type I errors (false convictions) accepting higher Type II error rates (wrongful acquittals).

Example: Decision Matrix for Hypothesis Testing

The following matrix shows the four possible outcomes in hypothesis testing, illustrating how Type I and Type II errors relate to the true state of the world and our test decision.