Generalized Additive Models

A regression can miss a pattern even when every predictor has a clear relationship with the response: a straight-line term cannot follow a sine-shaped association or a U-shaped one. Spline Regression gave us curves for an individual predictor. Combining several curves raises two further questions: how do we keep their contributions identifiable, and how do we choose their flexibility without judging them on the same observations used for fitting?

Generalized additive models (GAMs) fill that gap. They keep the additive structure of a linear model, replace each linear term with a smooth function, and let us pair that structure with the link functions familiar from generalized linear models. Formally, a GAM has the form

$$g(\mathbb{E}[y \mid \mathbf{x}]) = \beta_0 + f_1(x_1) + f_2(x_2) + \cdots + f_p(x_p),$$

where:

• $g$: the link function connecting the conditional mean to the additive predictor
• $\mathbb{E}[y \mid \mathbf{x}]$: the conditional mean of the response given the predictors
• $\beta_0$: the intercept
• $f_j$: a smooth function of predictor $x_j$
• $x_j$: the $j$-th predictor value, with $j=1,\ldots,p$ and $p$ predictors

Two questions guide the implementation:

• how to estimate the amount of flexibility (the smoothing parameter $\lambda$) without leaking information from validation into training,
• how to interpret a fitted additive model, including what it cannot capture (interactions).

This chapter develops those ideas from first principles. We will build a GAM by hand using spline bases and a difference penalty, show how the fit reduces to a closed-form ridge-style solve, extend the framework to non-Gaussian responses via a link function, and demonstrate the whole pipeline on a synthetic dataset where the true component functions are known. Implementation is done in NumPy for the mechanism and compared against statsmodels' GLMGam for a reference. We will close with practical guidance on when GAMs make sense and what to watch out for.