Generalized Additive Models › Example

The formula above is compact, but the algebra inside the penalized solve deserves to be seen once by hand. We now work through a tiny example where every matrix can be written out in full, using a deliberately small basis so the arithmetic stays readable. The lesson generalizes directly to cubic B-splines with many more knots; only the numbers change.

Suppose we have three points $x = (0.1, 0.5, 0.9)$ and responses $y = (1.2, 2.0, 1.5)$. On the domain $0 \le x \le 1$, place knots at $0$, $0.5$, and $1.0$, giving three linear hat functions:

$$h_1(x) = \max(0, 1 - 2x), \quad h_2(x) = 1 - |2x - 1|, \quad h_3(x) = \max(0, 2x - 1).$$

Evaluating these at our three $x$ values gives the $3 \times 3$ basis matrix

$$\mathbf{B}_0 = \begin{pmatrix} 0.8 & 0.2 & 0 \\ 0 & 1.0 & 0 \\ 0 & 0.2 & 0.8 \end{pmatrix}.$$

We will add an intercept column $\mathbf{1}$ as the first column, so the full design becomes

$$\mathbf{B} = \begin{pmatrix} 1 & 0.8 & 0.2 & 0 \\ 1 & 0 & 1.0 & 0 \\ 1 & 0 & 0.2 & 0.8 \end{pmatrix}.$$

The coefficient vector is $\boldsymbol\beta = (\beta_0, \beta_1, \beta_2, \beta_3)^\top$ with $\beta_0$ the intercept and $(\beta_1, \beta_2, \beta_3)$ giving the smooth.

The unpenalized normal-equation quantities are

$$\mathbf{B}^\top \mathbf{B} = \begin{pmatrix} 3 & 0.8 & 1.4 & 0.8 \\ 0.8 & 0.64 & 0.16 & 0 \\ 1.4 & 0.16 & 1.08 & 0.16 \\ 0.8 & 0 & 0.16 & 0.64 \end{pmatrix}, \qquad \mathbf{B}^\top \mathbf{y} = \begin{pmatrix} 4.7 \\ 0.96 \\ 2.54 \\ 1.20 \end{pmatrix}.$$

These come from the same rules as OLS: each entry of $\mathbf{B}^\top \mathbf{B}$ is a dot product of two columns of $\mathbf{B}$, and each entry of $\mathbf{B}^\top \mathbf{y}$ is a dot product of a column of $\mathbf{B}$ with the response. If we stopped here we would just be doing OLS on the basis expansion, flexible in principle, but with no safeguard against a wiggly fit. The penalty is what adds that safeguard.

The penalty comes next. Our smooth has three coefficients, so the second-difference matrix $\mathbf{D}$ has $3 - 2 = 1$ row, $\mathbf{D} = (1, -2, 1)$. The full penalty matrix has a zero row and column for the intercept (the intercept is not penalized) and $\mathbf{D}^\top \mathbf{D}$ in the smooth block:

$$\mathbf{P} = \begin{pmatrix} 0 & 0 & 0 & 0 \\ 0 & 1 & -2 & 1 \\ 0 & -2 & 4 & -2 \\ 0 & 1 & -2 & 1 \end{pmatrix}.$$

Take $\lambda = 0.1$. The penalized system is

$$\left( \mathbf{B}^\top \mathbf{B} + \lambda \mathbf{P} \right) \boldsymbol\beta = \mathbf{B}^\top \mathbf{y},$$

with

$$\mathbf{B}^\top \mathbf{B} + \lambda \mathbf{P} = \begin{pmatrix} 3 & 0.8 & 1.4 & 0.8 \\ 0.8 & 0.74 & -0.04 & 0.1 \\ 1.4 & -0.04 & 1.48 & -0.04 \\ 0.8 & 0.1 & -0.04 & 0.74 \end{pmatrix}.$$

Now compare the matrix above with the unpenalized $\mathbf{B}^\top \mathbf{B}$. The off-diagonal entries (the $-0.04$ and $0.1$ entries) are the change produced by the penalty. Those couplings come from second coefficient differences; they are the mechanism by which curvature is made expensive.

The displayed four-parameter system is still singular: the three hat functions sum to one, so their constant direction duplicates the intercept. Complete the fit by requiring the smooth to sum to zero at the three observations. Its constraint row is $\mathbf{c}^{\top}=(0,0.8,1.4,0.8)$. We solve the normal equations and that constraint together, using a Lagrange multiplier (an extra unknown that enforces the constraint).

The intercept is the sample response mean because the fitted smooth is centered. The fitted responses no longer interpolate all observations at this positive penalty.

If we now think about the extreme choices:

• With $\lambda = 0$ we recover the unpenalized fit, which on these three points fits every observation exactly (because the $3 \times 4$ design, once the smooth is constrained by identifiability, leaves enough freedom to interpolate all three points), including any noise.
• With $\lambda \to \infty$ the penalty forces $\beta_1 - 2\beta_2 + \beta_3 = 0$, meaning the fitted curve is exactly a straight line (a linear trend in the hat basis).
• For intermediate $\lambda$ (here $0.1$) the fit is a compromise that follows the data closely without allowing spurious curvature.

Both fits use a quadratic penalty, but a difference penalty couples neighboring coefficients, whereas ordinary ridge penalizes coefficient magnitudes. Their common normal-equation structure explains the connection. The next chapter Ridge Regularization (L2 Regularization) develops the ridge view of the penalty in more depth.

The worked example shows the arithmetic. The more important question is whether we can recover the sinusoid and the quadratic from the earlier synthetic data. That is what we implement next.