ANOVA (Analysis of Variance) › The Mathematics of One-Way ANOVA

Let's formalize these ideas mathematically. One-way ANOVA compares means across $k$ groups, where "one-way" means there's a single grouping factor.

• $k$ = number of groups
• $n_j$ = sample size of group $j$ (for $j = 1, 2, \ldots, k$)
• $N = \sum_{j=1}^{k} n_j$ = total sample size
• $x_{ij}$ = observation $i$ in group $j$
• $\bar{x}_j = \frac{1}{n_j}\sum_{i=1}^{n_j} x_{ij}$ = mean of group $j$
• $\bar{x} = \frac{1}{N}\sum_{j=1}^{k}\sum_{i=1}^{n_j} x_{ij}$ = grand mean (overall mean)

The total variation in the data can be decomposed into two components:

$$\underbrace{\sum_{j=1}^{k}\sum_{i=1}^{n_j}(x_{ij} - \bar{x})^2}_{SS_{Total}} = \underbrace{\sum_{j=1}^{k} n_j(\bar{x}_j - \bar{x})^2}_{SS_{Between}} + \underbrace{\sum_{j=1}^{k}\sum_{i=1}^{n_j}(x_{ij} - \bar{x}_j)^2}_{SS_{Within}}$$

Let's understand each component:

Total Sum of Squares ($SS_{Total}$): Measures total variation in the data. Each observation's squared deviation from the grand mean, summed across all observations. This is the same quantity we'd use to compute variance of the entire dataset.

Between-Group Sum of Squares ($SS_{Between}$): Measures variation of group means around the grand mean. Each group mean's squared deviation from the grand mean, weighted by sample size. Larger groups contribute more because their means are more reliable.

Within-Group Sum of Squares ($SS_{Within}$): Measures variation within each group. Each observation's squared deviation from its own group mean, summed across all groups. This represents the "noise" or baseline variability.

Sums of squares grow with sample size, so we convert them to mean squares (variance estimates) by dividing by degrees of freedom:

$$MS_{Between} = \frac{SS_{Between}}{k - 1} \quad \text{and} \quad MS_{Within} = \frac{SS_{Within}}{N - k}$$

Why these degrees of freedom?

• Between-groups ($k - 1$): We have $k$ group means, but they must average to the grand mean (one constraint), leaving $k - 1$ independent pieces of information.
• Within-groups ($N - k$): We have $N$ observations, but we estimate $k$ group means (one per group), losing $k$ degrees of freedom.

The F-statistic is the ratio of mean squares:

$$F = \frac{MS_{Between}}{MS_{Within}} = \frac{SS_{Between}/(k-1)}{SS_{Within}/(N-k)}$$

Under the null hypothesis that all population means are equal ($\mu_1 = \mu_2 = \cdots = \mu_k$), this F-statistic follows an F-distribution with $df_1 = k - 1$ and $df_2 = N - k$ degrees of freedom.

Intuition:

• $MS_{Within}$ estimates $\sigma^2$ (the common variance within groups) regardless of whether the null is true
• $MS_{Between}$ estimates $\sigma^2$ only if the null is true; if means differ, it estimates something larger
• If $H_0$ is true, $F \approx 1$; if $H_0$ is false, $F > 1$

Null hypothesis: $H_0: \mu_1 = \mu_2 = \cdots = \mu_k$ (all population means are equal)

Alternative hypothesis: $H_a:$ At least one population mean differs from the others

Note that rejecting $H_0$ tells us that some means differ, but not which ones. For that, we need post-hoc tests (discussed later).