Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Generalized Additive Models › Visualizing Generalized Additive Models

Michael BrenndoerferOctober 25, 20264 min read

Machine Learning from ScratchAll chapters

Part 5 of 15Linear and Generalized Linear Models

Chapter 39Generalized Additive Models › Visualizing Generalized Additive Models

Reading section 5 of 10 · Visualizing Generalized Additive Models · 4 minSection 5 of 10 · 4 min

Section 5 of 10 · 4 min read

Generalized Additive Models › Visualizing Generalized Additive ModelsSection view · Sections in this chapter5 / 10 ↓

Sections in this chapter10

Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Notes & commentsLeave notes and join the discussionSelection toolbarShow comment actions when selecting text. Turn off for the browser menu only.Comment indicatorsShow small counts beside annotated passagesPassage highlightsGently shade annotated text selections

Before we fit anything, we need to see the shape of the problem. Suppose we generate a dataset whose two predictors each have a known non-linear effect on a Gaussian response:

y_i = 1.0 + \sin(x_{i1}) + \left( 0.5 \, x_{i2}^2 - 0.5 \, x_{i2} \right) + \varepsilon_i, \qquad \varepsilon_i \sim \mathcal{N}(0, \sigma^2),

with $x_{i1}, x_{i2}$ drawn uniformly from $[-3, 3]$ and $\sigma = 0.4$ . The first component is a sinusoid; the second is a convex quadratic (upward-opening parabola). Both are clearly not linear, and the additive structure holds exactly because the two components enter as a sum.

PythonIn [7]
CollapseExpandimport matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import patsy
from book_plot_style import (
    PALETTE,
    polish_axes,
    seed_book_plots,
    use_book_style,
)
from scipy import stats

n = 400
rng = np.random.default_rng(0)
x1 = rng.uniform(-3.0, 3.0, n)
x2 = rng.uniform(-3.0, 3.0, n)
f1_true = np.sin(x1)
f2_true = 0.5 * x2**2 - 0.5 * x2
y = 1.0 + f1_true + f2_true + rng.normal(0.0, 0.4, n)
order1 = np.argsort(x1)
order2 = np.argsort(x2)
partial_y1 = y - f2_true - 1.0
partial_y2 = y - f1_true - 1.0
partial_y1_centered = partial_y1 - f1_true.mean()

Two quick plots showing each predictor against the response, with the true component overlaid, make the additive structure visible and give us a reference to check the fit against later.

VisualizationsOut [8]
CollapseExpand

Scatter of y minus the x2 contribution and intercept against x1 with a sine curve overlaid. — Response with the x2 contribution and intercept removed plotted against x1, alongside the true sine curve. The clear non-linearity shows what the spline fit must recover.

Scatter of y minus the x1 contribution and intercept against x2 with a quadratic curve overlaid. — Response with the x1 contribution and intercept removed plotted against x2, alongside the true upward-opening parabola. The curvature confirms that a linear term would miss real structure in this predictor.

These are oracle partial responses: the generating components are known only because the data are synthetic. In observed data, we cannot subtract the unknown true contribution of another predictor. The model instead estimates both components jointly; finite noisy samples and smoothing mean that recovery is approximate.

The other thing to visualize is the spline basis itself, since the smooth is a linear combination of these curves. With a modest number of basis functions, each is a localized bump, and the fitted smooth is determined by the weights we put on them.

PythonIn [9]
CollapseExpandgrid = np.linspace(-3.0, 3.0, 300)
B_grid = patsy.dmatrix(
    "0 + bs(grid, df=8, degree=3, include_intercept=True)",
    {"grid": grid},
    return_type="matrix",
)

VisualizationOut [10]
CollapseExpand

Eight locally supported cubic B-spline curves across the domain, including one-sided boundary curves. — A cubic B-spline basis with 8 basis functions evaluated on a regular grid. Each basis curve is a localized bump, so a weighted sum of these curves can produce a smooth function within the knot domain.

Each basis curve contributes only to a short segment of the $x$ range, and they overlap smoothly so that a weighted sum produces a smooth curve within the knot domain. Increasing the number of basis functions makes the space of possible curves richer. The job of $\lambda$ is to determine how aggressively we smooth the resulting weighted sum.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

Comments

Add to the conversation

Stay up to date