Ridge Regularization (L2 Regularization) › Summation Notation

As for LASSO, we can write the Ridge regression objective function as a summation:

$$\min_{\beta} \left\{ \sum_{i=1}^n (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^p \beta_j^2 \right\}$$

where:

• $y_i$ = actual (observed) value for observation $i$
• $\hat{y}_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_p x_{ip}$ = predicted value for observation $i$
• $\lambda$ = regularization parameter (non-negative; $\lambda \geq 0$)
• $\beta_0$ = intercept term (not penalized in Ridge regression)
• $\beta_j$ = coefficient for feature $j$ (where $j = 1, 2, \ldots, p$)
• $x_{ij}$ = value of feature $j$ for observation $i$
• $n$ = number of observations (sample size)
• $p$ = number of features (predictors)

This notation should look familiar from the previous section on LASSO. Rather than the L1 penalty, we have the L2 penalty, which is the sum of the squared coefficients.

$$\sum_{j=1}^p \beta_j^2 = \|\beta\|_2^2$$

This is the squared L2 norm (also called the L2 norm squared) of the coefficient vector $\boldsymbol{\beta}$, where $\|\boldsymbol{\beta}\|_2 = \sqrt{\sum_{j=1}^p \beta_j^2}$ is the L2 norm itself.

However, because matrix computations are much faster than summation, we can also write the Ridge regression objective function in matrix notation:

$$\min_{\beta} \left\{ ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||_2^2 + \lambda ||\boldsymbol{\beta}||_2^2 \right\}$$

where:

• $\mathbf{y}$ = $n \times 1$ vector of target values (observed responses)
• $\mathbf{X}$ = $n \times p$ design matrix of features (each row is an observation, each column is a feature)
• $\boldsymbol{\beta}$ = $p \times 1$ vector of coefficients (parameters to estimate)
• $||\cdot||_2$ = L2 norm (Euclidean norm), defined as $||\mathbf{v}||_2 = \sqrt{\sum_i v_i^2}$
• $||\cdot||_2^2$ = squared L2 norm, defined as $||\mathbf{v}||_2^2 = \sum_i v_i^2$

Because we are not encountering issues of non-differentiability, we can find the closed-form solution for the Ridge regression objective function. Taking the derivative with respect to $\boldsymbol{\beta}$ and setting it to zero:

$$\frac{\partial}{\partial \boldsymbol{\beta}} \left[ ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||_2^2 + \lambda ||\boldsymbol{\beta}||_2^2 \right] = 0$$

This yields the Ridge estimator:

$$\hat{\boldsymbol{\beta}}_{Ridge} = (\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I})^{-1} \mathbf{X}^T\mathbf{y}$$

where:

• $\hat{\boldsymbol{\beta}}_{Ridge}$ = Ridge regression coefficient estimates
• $\mathbf{X}^T$ = transpose of the design matrix
• $\mathbf{X}^T\mathbf{X}$ = $p \times p$ matrix (sum of squares and cross-products of features)
• $\lambda$ = regularization parameter
• $\mathbf{I}$ = $p \times p$ identity matrix (diagonal matrix with 1s on the diagonal and 0s elsewhere)
• $(\cdot)^{-1}$ = matrix inverse operation

• Bias-Variance Tradeoff: As $\lambda$ increases, bias increases but variance decreases
• Shrinkage: All coefficients are shrunk toward zero. In the special case of orthonormal features (where $\mathbf{X}^T\mathbf{X} = \mathbf{I}$), each coefficient is multiplied by the shrinkage factor $\frac{1}{1+\lambda}$, meaning $\hat{\beta}_{Ridge} = \frac{1}{1+\lambda}\hat{\beta}_{OLS}$
• Stability: The term $\lambda \mathbf{I}$ ensures $(\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I})$ is always invertible, even when $\mathbf{X}^T\mathbf{X}$ is singular (non-invertible)
• Regularization Path: As $\lambda \to 0$, Ridge approaches OLS ($\hat{\boldsymbol{\beta}}_{Ridge} \to \hat{\boldsymbol{\beta}}_{OLS}$); as $\lambda \to \infty$, all coefficients approach zero ($\hat{\boldsymbol{\beta}}_{Ridge} \to \mathbf{0}$)

This visualization illustrates the fundamental bias-variance tradeoff in Ridge regression. When λ is very small (left side), the model is close to OLS and may overfit the training data, resulting in low training error but high test error due to high variance. As λ increases, the training error rises because the model becomes more constrained (increased bias), but the test error initially decreases as variance is reduced. The optimal λ (green line) achieves the best balance, minimizing test error. Beyond this point, further increasing λ causes underfitting, where the model becomes too simple and both training and test errors increase due to excessive bias.

We skipped over the derivation of the closed-form solution for Ridge regression, but it is similar to the derivation for LASSO, but easier since we are not encountering issues of non-differentiability, making the math more straightforward.

To better understand how Ridge regression affects multiple coefficients simultaneously, let's visualize the regularization path.how each coefficient changes as we increase λ.