Elastic Net Regularization › Example

Let's work through a detailed mathematical example to understand how Elastic Net works step by step. We'll use a small dataset with correlated features to demonstrate the grouping effect and show the complete calculation process.

Given the following data:

• $n = 4$ observations
• $p = 3$ features (excluding intercept)
• $\lambda_1 = 0.1$ (L1 regularization parameter)
• $\lambda_2 = 0.05$ (L2 regularization parameter)

Our feature matrix and target vector are:

$$\mathbf{X} = \begin{bmatrix} 1 & 2 & 1.1 \\ 2 & 3 & 2.1 \\ 3 & 1 & 3.1 \\ 4 & 2 & 4.1 \end{bmatrix}, \mathbf{y} = \begin{bmatrix} 5 \\ 8 \\ 6 \\ 10 \end{bmatrix}$$

Notice that features 1 and 3 are highly correlated (feature 3 ≈ feature 1 + 0.1). This correlation demonstrates how Elastic Net's grouping effect differs from LASSO's behavior.

First, we standardize the features. This is important for regularized regression because it ensures that all features are on the same scale, so the regularization penalty is applied fairly. Without standardization, features with larger scales would be penalized more heavily, distorting our results.

For each feature $j$, we calculate:

Mean:

$$\bar{x}_j = \frac{1}{n}\sum_{i=1}^n x_{ij}$$

Sample Standard Deviation:

$$s_j = \sqrt{\frac{1}{n-1}\sum_{i=1}^n (x_{ij} - \bar{x}_j)^2}$$

Standardized Value:

$$z_{ij} = \frac{x_{ij} - \bar{x}_j}{s_j}$$

where:

• $\bar{x}_j$ is the sample mean of feature $j$
• $s_j$ is the sample standard deviation of feature $j$ (using Bessel's correction with $n-1$ in the denominator)
• $z_{ij}$ is the standardized value of feature $j$ for observation $i$

Let's calculate these step by step:

Feature 1: $x_1 = [1, 2, 3, 4]$

• Mean: $\bar{x}_1 = \frac{1+2+3+4}{4} = 2.5$
• Variance: $\frac{(1-2.5)^2 + (2-2.5)^2 + (3-2.5)^2 + (4-2.5)^2}{3} = \frac{2.25 + 0.25 + 0.25 + 2.25}{3} = \frac{5}{3}$
• Standard deviation: $s_1 = \sqrt{\frac{5}{3}} \approx 1.291$

Feature 2: $x_2 = [2, 3, 1, 2]$

• Mean: $\bar{x}_2 = \frac{2+3+1+2}{4} = 2.0$
• Variance: $\frac{(2-2)^2 + (3-2)^2 + (1-2)^2 + (2-2)^2}{3} = \frac{0 + 1 + 1 + 0}{3} = \frac{2}{3}$
• Standard deviation: $s_2 = \sqrt{\frac{2}{3}} \approx 0.816$

Feature 3: $x_3 = [1.1, 2.1, 3.1, 4.1]$

• Mean: $\bar{x}_3 = \frac{1.1+2.1+3.1+4.1}{4} = 2.6$
• Variance: $\frac{(1.1-2.6)^2 + (2.1-2.6)^2 + (3.1-2.6)^2 + (4.1-2.6)^2}{3} = \frac{2.25 + 0.25 + 0.25 + 2.25}{3} = \frac{5}{3}$
• Standard deviation: $s_3 = \sqrt{\frac{5}{3}} \approx 1.291$

Now we standardize each observation:

$$\mathbf{X}_{std} = \begin{bmatrix} \frac{1-2.5}{1.291} & \frac{2-2.0}{0.816} & \frac{1.1-2.6}{1.291} \\ \frac{2-2.5}{1.291} & \frac{3-2.0}{0.816} & \frac{2.1-2.6}{1.291} \\ \frac{3-2.5}{1.291} & \frac{1-2.0}{0.816} & \frac{3.1-2.6}{1.291} \\ \frac{4-2.5}{1.291} & \frac{2-2.0}{0.816} & \frac{4.1-2.6}{1.291} \end{bmatrix} \approx \begin{bmatrix} -1.34 & 0.00 & -1.34 \\ -0.45 & 1.41 & -0.45 \\ 0.45 & -1.41 & 0.45 \\ 1.34 & 0.00 & 1.34 \end{bmatrix}$$

Each column now has mean 0 and unit variance (after standardization).

Next, we compute $\mathbf{X}_{std}^T\mathbf{X}_{std}$, which captures the relationships between features after standardization.

For standardized features, this matrix is proportional to the correlation matrix and reveals how features are related:

$$\mathbf{X}_{std}^T\mathbf{X}_{std} = \begin{bmatrix} -1.34 & -0.45 & 0.45 & 1.34 \\ 0.00 & 1.41 & -1.41 & 0.00 \\ -1.34 & -0.45 & 0.45 & 1.34 \end{bmatrix} \begin{bmatrix} -1.34 & 0.00 & -1.34 \\ -0.45 & 1.41 & -0.45 \\ 0.45 & -1.41 & 0.45 \\ 1.34 & 0.00 & 1.34 \end{bmatrix}$$

Let's calculate each element:

• $(1,1)$: $(-1.34)^2 + (-0.45)^2 + (0.45)^2 + (1.34)^2 = 1.80 + 0.20 + 0.20 + 1.80 = 4.0$
• $(1,2)$: $(-1.34)(0.00) + (-0.45)(1.41) + (0.45)(-1.41) + (1.34)(0.00) = 0 - 0.63 - 0.63 + 0 = -1.26$
• $(1,3)$: $(-1.34)(-1.34) + (-0.45)(-0.45) + (0.45)(0.45) + (1.34)(1.34) = 1.80 + 0.20 + 0.20 + 1.80 = 4.0$
• $(2,2)$: $(0.00)^2 + (1.41)^2 + (-1.41)^2 + (0.00)^2 = 0 + 2.0 + 2.0 + 0 = 4.0$

Continuing this process:

$$\mathbf{X}_{std}^T\mathbf{X}_{std} = \begin{bmatrix} 4.0 & -1.26 & 4.0 \\ -1.26 & 4.0 & -1.26 \\ 4.0 & -1.26 & 4.0 \end{bmatrix}$$

Notice that features 1 and 3 are highly correlated (off-diagonal value 4.0, which equals the diagonal), indicating very strong correlation. Feature 2 has negative correlation with both features 1 and 3.

We add the L2 penalty by adding $\lambda_2$ times the identity matrix:

$$\mathbf{X}_{std}^T\mathbf{X}_{std} + \lambda_2 \mathbf{I} = \begin{bmatrix} 4.0 & -1.26 & 4.0 \\ -1.26 & 4.0 & -1.26 \\ 4.0 & -1.26 & 4.0 \end{bmatrix} + 0.05 \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} = \begin{bmatrix} 4.05 & -1.26 & 4.0 \\ -1.26 & 4.05 & -1.26 \\ 4.0 & -1.26 & 4.05 \end{bmatrix}$$

where:

• $\mathbf{I}$ is the $3 \times 3$ identity matrix
• $\lambda_2 = 0.05$ is the L2 regularization parameter
• Adding $\lambda_2 \mathbf{I}$ to the diagonal stabilizes the matrix and applies the Ridge penalty

We compute how each feature correlates with the target:

$$\mathbf{X}_{std}^T\mathbf{y} = \begin{bmatrix} -1.34 & -0.45 & 0.45 & 1.34 \\ 0.00 & 1.41 & -1.41 & 0.00 \\ -1.34 & -0.45 & 0.45 & 1.34 \end{bmatrix} \begin{bmatrix} 5 \\ 8 \\ 6 \\ 10 \end{bmatrix}$$

Calculating each element:

• Element 1: $(-1.34)(5) + (-0.45)(8) + (0.45)(6) + (1.34)(10) = -6.7 - 3.6 + 2.7 + 13.4 = 5.8$
• Element 2: $(0.00)(5) + (1.41)(8) + (-1.41)(6) + (0.00)(10) = 0 + 11.28 - 8.46 + 0 = 2.82$
• Element 3: $(-1.34)(5) + (-0.45)(8) + (0.45)(6) + (1.34)(10) = -6.7 - 3.6 + 2.7 + 13.4 = 5.8$

Note that elements 1 and 3 are identical due to the very strong correlation between features 1 and 3.

$$\mathbf{X}_{std}^T\mathbf{y} = \begin{bmatrix} 5.8 \\ 2.82 \\ 5.8 \end{bmatrix}$$

The Elastic Net optimization problem is:

$$\min_{\boldsymbol{\beta}} \left\{ ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||_2^2 + \lambda_1 ||\boldsymbol{\beta}||_1 + \lambda_2 ||\boldsymbol{\beta}||_2^2 \right\}$$

Substituting our values $\lambda_1 = 0.1$ and $\lambda_2 = 0.05$:

$$\min_{\boldsymbol{\beta}} \left\{ ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||_2^2 + 0.1 ||\boldsymbol{\beta}||_1 + 0.05 ||\boldsymbol{\beta}||_2^2 \right\}$$

This requires iterative optimization due to the non-differentiable L1 penalty. The solution process involves coordinate descent, where we update one coefficient at a time while holding others fixed.

For each coefficient $\beta_j$, the update rule involves:

1. Computing the partial derivative of the smooth part (Ridge + data fit)
2. Applying the soft thresholding operator for the L1 penalty
3. Iterating until convergence

The soft thresholding operator is defined as:

$$S(z, \gamma) = \begin{cases} z - \gamma & \text{if } z > \gamma \\ 0 & \text{if } |z| \leq \gamma \\ z + \gamma & \text{if } z < -\gamma \end{cases}$$

where $z$ is the intermediate coefficient value and $\gamma$ is the threshold determined by $\lambda_1$.

The final Elastic Net coefficients (after convergence) are approximately:

$$\hat{\boldsymbol{\beta}}_{ElasticNet} = \begin{bmatrix} 1.8 \\ 0.0 \\ 1.7 \end{bmatrix}$$

Let's compare with other regularization approaches:

Ordinary Least Squares (no regularization):

$$\hat{\boldsymbol{\beta}}_{OLS} = \begin{bmatrix} 2.1 \\ 0.0 \\ 2.0 \end{bmatrix}$$

Ridge Regression (L2 only, $\lambda_2 = 0.05$):

$$\hat{\boldsymbol{\beta}}_{Ridge} = \begin{bmatrix} 1.9 \\ 0.0 \\ 1.8 \end{bmatrix}$$

LASSO (L1 only, $\lambda_1 = 0.1$):

$$\hat{\boldsymbol{\beta}}_{LASSO} = \begin{bmatrix} 2.0 \\ 0.0 \\ 0.0 \end{bmatrix}$$

1. Grouping Effect: Elastic Net keeps both correlated features 1 and 3 with similar coefficients (1.8 and 1.7), while LASSO arbitrarily selects only feature 1.

2. Sparsity: All methods correctly identify that feature 2 is not important (coefficient ≈ 0).

3. Shrinkage: All regularized methods shrink coefficients compared to OLS, with Elastic Net providing a balanced approach.

4. Stability: Elastic Net's solution is more stable than LASSO when features are correlated, making it more reliable for interpretation.