Generalized Additive Models › Summary

A generalized additive model is the natural upgrade of linear and spline regression when multiple predictors each have non-linear effects but no interaction structure. The additive predictor $\eta = \beta_0 + \sum_j f_j(x_j)$ is connected to the response mean via a link function $g$, allowing the same machinery to handle Gaussian, binary, count, and other response types. Each smooth $f_j$ is expanded in a spline basis, and a difference penalty regularizes how wiggly the resulting curve can be, turning the Gaussian identity-link fit into a penalized least-squares problem. After identifiability constraints, an invertible penalized system has the solution

$$\hat{\boldsymbol\beta} = \left( \mathbf{B}^\top \mathbf{B} + \mathbf{S} \right)^{-1} \mathbf{B}^\top \mathbf{y},$$

which is structurally identical to ridge regression on an expanded feature space. For non-identity links, a weighted penalized system reappears inside an IRLS loop for suitable non-Gaussian GLMs.

The worked procedure selects a shared smoothing parameter by cross-validation within training data, fitting the basis and centering inside every fold, then scores once on independent synthetic test observations. Other smoothness-selection criteria answer related questions under their own assumptions. Interpretation is done by plotting each component separately after centering it to have zero mean, and reading off the effect on the scale of the link function. The additive restriction excludes interactions unless they are included explicitly. When interactions are suspected, tensor-product smooths can be added at the cost of sample size and interpretability.

A GAM extends a GLM by allowing smooth terms in its predictor. It contains a linear GLM when those terms are linear, and its component curves expose the fitted additive associations. It is a good first model to try whenever a linear regression or a linear GLM leaves obvious non-linear residuals. Its Gaussian fitting mechanism, quadratic penalized least squares on an expanded basis, is closely related to ridge regression. The next chapter Ridge Regularization (L2 Regularization) develops that connection further. Lasso instead uses an absolute-value penalty and requires a different optimization problem.