Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

The F-Test and F-Distribution › F-Test in Regression: Overall Model Significance

Michael BrenndoerferJanuary 4, 20265 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 16The F-Test and F-Distribution › F-Test in Regression: Overall Model Significance

Reading section 5 of 10 · F-Test in Regression: Overall Model Significance · 5 minSection 5 of 10 · 5 min

Section 5 of 10 · 5 min read

The F-Test and F-Distribution › F-Test in Regression: Overall Model SignificanceOne section · Sections in this chapter5 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Beyond comparing two variances, the F-test plays a central role in regression analysis. When you fit a regression model, an immediate question is: does this model explain any variation at all?

The ANOVA DecompositionLink Copied

Every regression model partitions the total variation in the response variable into two components:

SS_{Total} = SS_{Regression} + SS_{Residual}

where:

$SS_{Total}$ = $\sum_{i=1}^{n}(y_i - \bar{y})^2$ : Total variation in $y$
$SS_{Regression}$ = $\sum_{i=1}^{n}(\hat{y}_i - \bar{y})^2$ : Variation explained by the model
$SS_{Residual}$ = $\sum_{i=1}^{n}(y_i - \hat{y}_i)^2$ : Variation left unexplained

The F-test asks: is the explained variation large enough, relative to the unexplained variation, to conclude that the model has real predictive value?

The F-Statistic for RegressionLink Copied

Test statistic:

F = \frac{SS_{Regression} / p}{SS_{Residual} / (n - p - 1)} = \frac{MS_{Regression}}{MS_{Residual}}

where:

$p$ = number of predictors (excluding intercept)
$n$ = number of observations
$MS$ = "Mean Square" = sum of squares divided by degrees of freedom

Degrees of freedom:

Numerator: $df_1 = p$ (one for each predictor)
Denominator: $df_2 = n - p - 1$ (residual degrees of freedom)

Null hypothesis: $H_0: \beta_1 = \beta_2 = \cdots = \beta_p = 0$ (all predictor coefficients are zero)

If the null is true, none of the predictors help explain $y$ , and $MS_{Regression}$ should be similar to $MS_{Residual}$ , giving $F \approx 1$ . A large F indicates the model explains more variation than expected by chance.

Worked Example: Multiple RegressionLink Copied

A researcher studies factors affecting house prices, fitting a model with square footage and number of bedrooms as predictors.

In[8]:

Code

import numpy as np
from scipy import stats

# Generate housing data
np.random.seed(42)
n = 50

sqft = np.random.uniform(1000, 3000, n)
bedrooms = np.random.randint(2, 6, n)
price = 50000 + 150 * sqft + 20000 * bedrooms + np.random.normal(0, 30000, n)

# Fit regression using linear algebra
X = np.column_stack([np.ones(n), sqft, bedrooms])
beta = np.linalg.lstsq(X, price, rcond=None)[0]
y_pred = X @ beta

# Calculate sums of squares
y_bar = np.mean(price)
ss_total = np.sum((price - y_bar) ** 2)
ss_residual = np.sum((price - y_pred) ** 2)
ss_regression = ss_total - ss_residual

# Degrees of freedom
p = 2  # number of predictors (sqft and bedrooms)
df_regression = p
df_residual = n - p - 1
df_total = n - 1

# Mean squares
ms_regression = ss_regression / df_regression
ms_residual = ss_residual / df_residual

# F-statistic and p-value
f_stat = ms_regression / ms_residual
p_value = stats.f.sf(f_stat, df_regression, df_residual)

# R-squared
r_squared = ss_regression / ss_total
adj_r_squared = 1 - (1 - r_squared) * (n - 1) / (n - p - 1)

Out[9]:

Console

F-Test for Regression: House Price Model
=================================================================

Model: Price = beta_0 + beta_1 * SquareFootage + beta_2 * Bedrooms + epsilon

Fitted coefficients:
  Intercept (beta_0):     $64,173
  Square footage (beta_1): $147.43 per sqft
  Bedrooms (beta_2):       $18,284 per bedroom

ANOVA Table:
-----------------------------------------------------------------
Source                   SS     df             MS          F            p
-----------------------------------------------------------------
Regression   399,645,915,189      2 199,822,957,594     224.36     9.04e-25
Residual     41,860,284,034     47    890,644,341
Total        441,506,199,223     49
-----------------------------------------------------------------

R-squared:          0.9052
Adjusted R-squared: 0.9012

Interpretation: The model explains 90.5% of the variance in house prices.
The F-test (F = 224.4, p < 0.001) confirms this is statistically significant.

Out[10]:

Visualization

Scatter plot of actual vs predicted house prices with perfect prediction line. — Actual vs. predicted house prices.

Bar chart showing total variance decomposed into explained and unexplained components. — ANOVA decomposition of total variance into explained (regression) and unexplained (residual) components.

Connection to R-SquaredLink Copied

The F-statistic is directly related to $R^2$ :

F = \frac{R^2 / p}{(1 - R^2) / (n - p - 1)}

This shows that:

Higher $R^2$ → larger F → smaller p-value
More predictors ( $p$ ) → penalizes the F-statistic (need more explained variance to be significant)
Larger sample size ( $n$ ) → more power to detect small $R^2$ as significant