Ridge Regularization (L2 Regularization) › Example

Let's work through a simple mathematical example to understand how Ridge regression works. We remind ourselves of the ridge estimator:

$$\hat{\boldsymbol{\beta}}_{Ridge} = (\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I})^{-1} \mathbf{X}^T\mathbf{y}$$

We'll use a small dataset with 3 observations and 2 features (excluding scaling to make the example easier to follow). Given following data:

• $n = 3$ observations
• $p = 2$ features
• $\lambda = 1$ (regularization parameter)

$$\mathbf{X} = \begin{bmatrix} 1 & 2 \\ 2 & 3 \\ 3 & 1 \end{bmatrix}, \mathbf{y} = \begin{bmatrix} 5 \\ 8 \\ 6 \end{bmatrix}$$

Let's walk through the Ridge regression calculation step by step.

Step 1: Calculate $\mathbf{X}^T\mathbf{X}$
First, we compute the matrix multiplication of the transpose of $\mathbf{X}$ and $\mathbf{X}$. This gives us the sum of squares and cross-products matrix of the features, which is used in both OLS and Ridge regression.

$$\mathbf{X}^T\mathbf{X} = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 3 & 1 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 2 & 3 \\ 3 & 1 \end{bmatrix} = \begin{bmatrix} 14 & 11 \\ 11 & 14 \end{bmatrix}$$

To verify: The $(1,1)$ element is $1^2 + 2^2 + 3^2 = 1 + 4 + 9 = 14$. The $(1,2)$ element is $1 \cdot 2 + 2 \cdot 3 + 3 \cdot 1 = 2 + 6 + 3 = 11$.

Note: This matrix is symmetric, meaning $X_{ij} = X_{ji}$ for all $i, j$. This is always true for $\mathbf{X}^T\mathbf{X}$.

Step 2: Add regularization term $\lambda \mathbf{I}$
Next, we add the regularization term $\lambda \mathbf{I}$ to the matrix. This penalizes large coefficients and helps prevent overfitting by making the matrix more stable. The resulting matrix is better conditioned, a term from numerical linear algebra that means small changes in input tend to produce small changes in output.

$$\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I} = \begin{bmatrix} 14 & 11 \\ 11 & 14 \end{bmatrix} + \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 15 & 11 \\ 11 & 15 \end{bmatrix}$$

Step 3: Calculate $\mathbf{X}^T\mathbf{y}$
We then compute the product of the transpose of $\mathbf{X}$ and the target vector $\mathbf{y}$. This operation measures how each feature in $\mathbf{X}$ is correlated with the target variable $\mathbf{y}$. It forms the right-hand side of the normal equations used in both OLS and Ridge regression.

$$\mathbf{X}^T\mathbf{y} = \begin{bmatrix} 1 & 2 & 3 \\ 2 & 3 & 1 \end{bmatrix} \begin{bmatrix} 5 \\ 8 \\ 6 \end{bmatrix} = \begin{bmatrix} 39 \\ 40 \end{bmatrix}$$

To verify: The first element is $1 \cdot 5 + 2 \cdot 8 + 3 \cdot 6 = 5 + 16 + 18 = 39$. The second element is $2 \cdot 5 + 3 \cdot 8 + 1 \cdot 6 = 10 + 24 + 6 = 40$.

Step 4: Find the inverse and solve
Now, we invert the regularized matrix from Step 2 to solve the linear system and find the coefficients.

$$(\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I})^{-1} = \frac{1}{15 \cdot 15 - 11 \cdot 11} \begin{bmatrix} 15 & -11 \\ -11 & 15 \end{bmatrix} = \frac{1}{104} \begin{bmatrix} 15 & -11 \\ -11 & 15 \end{bmatrix}$$

Note: In mathematics, the inverse of a $2 \times 2$ matrix $\begin{bmatrix} a & b \\ c & d \end{bmatrix}$ is given by $\frac{1}{ad-bc} \begin{bmatrix} d & -b \\ -c & a \end{bmatrix}$, provided $ad-bc \neq 0$. For larger matrices, the process involves more advanced linear algebra techniques such as Gaussian elimination or using determinants and adjugates.

In practice, when working with code, you can invert matrices using libraries like NumPy. For example, in Python, you can use np.linalg.inv(matrix) to compute the inverse of a matrix. However, for solving systems like Ridge regression, it's often more numerically stable to use np.linalg.solve() or specialized solvers rather than explicitly computing the inverse.

np.linalg.inv(A) explicitly computes the inverse of a matrix and then multiplies it by $b$, which is slower and can amplify numerical errors. In contrast, np.linalg.solve(A, b) directly solves the system $A x = b$ using efficient factorizations without forming the inverse, making it both faster and more numerically stable.

Step 5: Calculate Ridge coefficients
Finally, we multiply the inverse matrix by $\mathbf{X}^T\mathbf{y}$ to obtain the Ridge regression coefficients. These coefficients are shrunk compared to the OLS solution due to the regularization.

$$\hat{\boldsymbol{\beta}}_{Ridge} = \frac{1}{104} \begin{bmatrix} 15 & -11 \\ -11 & 15 \end{bmatrix} \begin{bmatrix} 39 \\ 40 \end{bmatrix} = \frac{1}{104} \begin{bmatrix} 145 \\ 171 \end{bmatrix} = \begin{bmatrix} 1.394 \\ 1.644 \end{bmatrix}$$

To verify the matrix multiplication:

• First element: $15 \cdot 39 + (-11) \cdot 40 = 585 - 440 = 145$
• Second element: $(-11) \cdot 39 + 15 \cdot 40 = -429 + 600 = 171$

Then dividing by 104: $145/104 \approx 1.394$ and $171/104 \approx 1.644$.

If we compare it with OLS which has no regularization ($\lambda = 0$), the OLS solution would be:

$$\hat{\boldsymbol{\beta}}_{OLS} = (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{X}^T\mathbf{y}$$

Let's calculate this step by step:

First, compute the determinant: $14 \cdot 14 - 11 \cdot 11 = 196 - 121 = 75$

$$(\mathbf{X}^T\mathbf{X})^{-1} = \frac{1}{75} \begin{bmatrix} 14 & -11 \\ -11 & 14 \end{bmatrix}$$

Then multiply by $\mathbf{X}^T\mathbf{y}$:

$$\hat{\boldsymbol{\beta}}_{OLS} = \frac{1}{75} \begin{bmatrix} 14 & -11 \\ -11 & 14 \end{bmatrix} \begin{bmatrix} 39 \\ 40 \end{bmatrix} = \frac{1}{75} \begin{bmatrix} 106 \\ 131 \end{bmatrix} = \begin{bmatrix} 1.413 \\ 1.747 \end{bmatrix}$$

where $14 \cdot 39 - 11 \cdot 40 = 546 - 440 = 106$ and $-11 \cdot 39 + 14 \cdot 40 = -429 + 560 = 131$.

Comparison:

• Ridge coefficients: $\begin{bmatrix} 1.394 \\ 1.644 \end{bmatrix}$
• OLS coefficients: $\begin{bmatrix} 1.413 \\ 1.747 \end{bmatrix}$

Notice how Ridge shrunk both coefficients toward zero compared to OLS. The first coefficient decreased from 1.413 to 1.394, and the second from 1.747 to 1.644.