Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

The F-Test and F-Distribution › Comparing Nested Models

Michael BrenndoerferJanuary 4, 20264 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 16The F-Test and F-Distribution › Comparing Nested Models

Reading section 6 of 10 · Comparing Nested Models · 4 minSection 6 of 10 · 4 min

Section 6 of 10 · 4 min read

The F-Test and F-Distribution › Comparing Nested ModelsOne section · Sections in this chapter6 / 10 ↓

Sections in this chapter10

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One of the most powerful applications of the F-test is comparing nested regression models. A model is "nested" within another if it's a special case obtained by setting some coefficients to zero.

Question: Does adding extra predictors significantly improve the model?

The Nested Model F-TestLink Copied

Consider:

Reduced model: $y = \beta_0 + \beta_1 x_1 + \epsilon$ (1 predictor)
Full model: $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \epsilon$ (3 predictors)

The reduced model is nested within the full model (set $\beta_2 = \beta_3 = 0$ ).

Test statistic:

F = \frac{(SS_{res,reduced} - SS_{res,full}) / (df_{reduced} - df_{full})}{SS_{res,full} / df_{full}}

Interpretation:

Numerator: Improvement (reduction in residual SS) per additional predictor
Denominator: Unexplained variance per degree of freedom in the full model
$df_{reduced} - df_{full}$ = number of additional predictors being tested

Null hypothesis: The additional predictors have no effect ( $\beta_2 = \beta_3 = 0$ )

Worked Example: Model SelectionLink Copied

A researcher wants to know if adding interaction terms improves a base model.

In[11]:

Code

import numpy as np
from scipy import stats


def compare_nested_models(y, X_reduced, X_full):
    """
    Compare nested regression models using F-test.

    Returns F-statistic, p-value, and improvement in R-squared.
    """
    n = len(y)

    # Fit both models
    beta_reduced = np.linalg.lstsq(X_reduced, y, rcond=None)[0]
    beta_full = np.linalg.lstsq(X_full, y, rcond=None)[0]

    y_pred_reduced = X_reduced @ beta_reduced
    y_pred_full = X_full @ beta_full

    # Residual sums of squares
    ss_res_reduced = np.sum((y - y_pred_reduced) ** 2)
    ss_res_full = np.sum((y - y_pred_full) ** 2)
    ss_total = np.sum((y - np.mean(y)) ** 2)

    # R-squared for each model
    r2_reduced = 1 - ss_res_reduced / ss_total
    r2_full = 1 - ss_res_full / ss_total

    # Degrees of freedom
    p_reduced = X_reduced.shape[1] - 1
    p_full = X_full.shape[1] - 1

    df_reduced = n - p_reduced - 1
    df_full = n - p_full - 1
    df_diff = df_reduced - df_full  # = number of added predictors

    # F-statistic
    f_stat = ((ss_res_reduced - ss_res_full) / df_diff) / (
        ss_res_full / df_full
    )
    p_value = stats.f.sf(f_stat, df_diff, df_full)

    return {
        "f_stat": f_stat,
        "df_num": df_diff,
        "df_denom": df_full,
        "p_value": p_value,
        "r2_reduced": r2_reduced,
        "r2_full": r2_full,
        "r2_change": r2_full - r2_reduced,
    }


# Example: Testing whether x2 and interaction add value
np.random.seed(42)
n = 60
x1 = np.random.normal(0, 1, n)
x2 = np.random.normal(0, 1, n)
x3_noise = np.random.normal(0, 1, n)  # Pure noise

# True model: y depends on x1 and x2, with interaction
y = 5 + 3 * x1 + 2 * x2 + 1.5 * x1 * x2 + np.random.normal(0, 1, n)

# Define nested models
X_base = np.column_stack([np.ones(n), x1])
X_with_x2 = np.column_stack([np.ones(n), x1, x2])
X_full = np.column_stack([np.ones(n), x1, x2, x1 * x2])
X_with_noise = np.column_stack([np.ones(n), x1, x2, x1 * x2, x3_noise])

# Compare models
result1 = compare_nested_models(y, X_base, X_with_x2)
result2 = compare_nested_models(y, X_with_x2, X_full)
result3 = compare_nested_models(y, X_full, X_with_noise)

Out[12]:

Console

Nested Model Comparisons Using F-Test
======================================================================

True model: y = 5 + 3*x1 + 2*x2 + 1.5*x1*x2 + noise

Comparison 1: Base model (x1 only) vs. adding x2
----------------------------------------------------------------------
  R² change:   0.6016 -> 0.7688 (+0.1672)
  F(1, 57) = 41.22, p = 2.93e-08
  Conclusion:  Adding x2 significantly improves model

Comparison 2: Model with x1, x2 vs. adding interaction x1*x2
----------------------------------------------------------------------
  R² change:   0.7688 -> 0.9221 (+0.1534)
  F(1, 56) = 110.26, p = 7.56e-15
  Conclusion:  Adding interaction significantly improves model

Comparison 3: Full model vs. adding noise variable x3
----------------------------------------------------------------------
  R² change:   0.9221 -> 0.9222 (+0.0001)
  F(1, 55) = 0.08, p = 0.7776
  Conclusion:  x3 (noise) does not improve model, as expected

The F-test correctly identifies:

Adding $x_2$ (true predictor) significantly improves the model
Adding the interaction $x_1 \cdot x_2$ (true effect) significantly improves the model
Adding $x_3$ (pure noise) does not significantly improve the model

When to Use Nested Model ComparisonLink Copied

This technique is useful for:

Variable selection: Testing whether specific predictors should be included
Testing interactions: Does an interaction term add explanatory power?
Polynomial terms: Does a quadratic term improve on a linear model?
Group comparisons: Testing whether different groups need different slopes (via interaction with a group indicator)