Elastic Net Regularization › Formula

Let's build up the Elastic Net objective function step by step, starting from the most intuitive form and explaining each mathematical transformation along the way.

We begin with the standard multiple linear regression problem. Our goal is to find coefficients $\beta_0, \beta_1, \beta_2, \ldots, \beta_p$ that minimize the sum of squared errors between our predictions and the actual values:

$$\text{SSE} = \sum_{i=1}^n (y_i - \hat{y}_i)^2$$

where:

• $y_i$ is the actual target value for observation $i$ (where $i = 1, 2, \ldots, n$)
• $\hat{y}_i$ is the predicted value for observation $i$
• $n$ is the number of observations in the dataset

Here, $\hat{y}_i$ represents our predicted value for observation $i$, which we calculate as:

$$\hat{y}_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_p x_{ip}$$

where:

• $\beta_0$ is the intercept term (constant offset)
• $\beta_j$ is the coefficient (weight) for feature $j$ (where $j = 1, 2, \ldots, p$)
• $x_{ij}$ is the value of feature $j$ for observation $i$
• $p$ is the number of features (predictors) in the model

Now, instead of just minimizing the sum of squared errors, we want to add penalty terms that will help us control the complexity of our model. Elastic Net adds two types of penalties:

1. L1 Penalty (LASSO component): $\lambda_1 \sum_{j=1}^p |\beta_j|$
2. L2 Penalty (Ridge component): $\lambda_2 \sum_{j=1}^p \beta_j^2$

where:

• $\lambda_1 > 0$ is the L1 regularization parameter (controls the strength of the L1 penalty)
• $\lambda_2 > 0$ is the L2 regularization parameter (controls the strength of the L2 penalty)
• $|\beta_j|$ is the absolute value of coefficient $\beta_j$
• $\beta_j^2$ is the squared value of coefficient $\beta_j$

Let's understand why we use these specific penalty forms:

Why the L1 penalty uses absolute values?

The absolute value function $|\beta_j|$ has a special property: its derivative is not continuous at zero. This creates a "corner" in the objective function that can drive coefficients exactly to zero, effectively performing automatic feature selection. When we take the derivative of $|\beta_j|$, we get:

• $\frac{d}{d\beta_j}|\beta_j| = 1$ when $\beta_j > 0$
• $\frac{d}{d\beta_j}|\beta_j| = -1$ when $\beta_j < 0$
• $\frac{d}{d\beta_j}|\beta_j|$ is undefined at $\beta_j = 0$

This discontinuity is what allows LASSO to set coefficients exactly to zero.

Why the L2 penalty uses squared terms?

The squared penalty $\beta_j^2$ has a smooth, continuous derivative everywhere: $\frac{d}{d\beta_j}\beta_j^2 = 2\beta_j$. This smoothness helps with correlated features by encouraging similar coefficients for similar features, creating a "grouping effect." The penalty grows quadratically, so larger coefficients are penalized more heavily than smaller ones.

Combining all three components, our Elastic Net objective function becomes:

$$\min_{\boldsymbol{\beta}} \left\{ \sum_{i=1}^n (y_i - \hat{y}_i)^2 + \lambda_1 \sum_{j=1}^p |\beta_j| + \lambda_2 \sum_{j=1}^p \beta_j^2 \right\}$$

where:

• $\boldsymbol{\beta} = [\beta_0, \beta_1, \ldots, \beta_p]^T$ is the vector of coefficients we are optimizing
• $\sum_{i=1}^n (y_i - \hat{y}_i)^2$ is the sum of squared errors (data fit term)
• $\lambda_1 \sum_{j=1}^p |\beta_j|$ is the L1 penalty term (encourages sparsity)
• $\lambda_2 \sum_{j=1}^p \beta_j^2$ is the L2 penalty term (encourages small coefficients)

Let's break down what each part does:

• Data Fit Term $\sum_{i=1}^n (y_i - \hat{y}_i)^2$: Ensures our predictions are close to the actual values
• L1 Penalty $\lambda_1 \sum_{j=1}^p |\beta_j|$: Encourages sparsity by driving some coefficients to exactly zero
• L2 Penalty $\lambda_2 \sum_{j=1}^p \beta_j^2$: Encourages small coefficients and handles correlated features well

The parameters $\lambda_1$ and $\lambda_2$ control the strength of each penalty:

• When $\lambda_1 = 0$ and $\lambda_2 > 0$: We get pure Ridge regression
• When $\lambda_1 > 0$ and $\lambda_2 = 0$: We get pure LASSO regression
• When $\lambda_1 > 0$ and $\lambda_2 > 0$: We get Elastic Net

We can write the penalty terms more compactly using norm notation:

$$\lambda_1 \sum_{j=1}^p |\beta_j| + \lambda_2 \sum_{j=1}^p \beta_j^2 = \lambda_1 \|\boldsymbol{\beta}\|_1 + \lambda_2 \|\boldsymbol{\beta}\|_2^2$$

where:

• $\|\boldsymbol{\beta}\|_1 = \sum_{j=1}^p |\beta_j|$ is the L1 norm (Manhattan distance)
• $\|\boldsymbol{\beta}\|_2 = \sqrt{\sum_{j=1}^p \beta_j^2}$ is the L2 norm (Euclidean distance)
• $\|\boldsymbol{\beta}\|_2^2 = \sum_{j=1}^p \beta_j^2$ is the squared L2 norm

The L1 norm measures the sum of absolute values, while the L2 norm measures the square root of the sum of squares. In our formulation, we use the squared L2 norm ($\|\boldsymbol{\beta}\|_2^2$) for mathematical convenience.

Elastic Net's effectiveness comes from how these two penalties complement each other:

1. The L1 penalty provides sparsity (feature selection) but can be unstable with correlated features
2. The L2 penalty provides stability with correlated features but doesn't perform feature selection
3. Together, they provide both sparsity and stability

This is particularly valuable when we have many features, some of which are correlated, and we want both interpretability through feature selection and stable coefficient estimates.

To understand why Elastic Net combines the best of both worlds, let's visualize the constraint regions geometrically. In regularized regression, we can think of the problem as minimizing the sum of squared errors subject to a constraint on the coefficients.

This geometric view reveals several key insights:

1. L1 Constraint (Diamond): The sharp corners at the axes mean that the loss contours are likely to first touch the constraint region at a corner, resulting in sparse solutions where some coefficients are exactly zero.

2. L2 Constraint (Circle): The smooth, round shape means the loss contours will touch the constraint region at a point where all coefficients are typically non-zero but small. This provides stability but no sparsity.

3. Elastic Net Constraint (Rounded Diamond): By combining both penalties, we get a shape that has both corners (for sparsity) and smooth edges (for stability). This allows the model to achieve sparsity when needed while maintaining the grouping effect for correlated features.

The red contours represent levels of the loss function (sum of squared errors). The optimal solution for each method occurs where these contours first touch the respective constraint region. This geometric interpretation makes it clear why Elastic Net can achieve both feature selection and stability.it literally combines the geometric properties of both L1 and L2 constraints.

Now let's translate our objective function into matrix notation, which is more compact and reveals the mathematical structure more clearly.

First, let's define our matrices:

• $\mathbf{y}$ is an $n \times 1$ vector containing all target values: $\mathbf{y} = [y_1, y_2, \ldots, y_n]^T$
• $\mathbf{X}$ is an $n \times (p+1)$ design matrix where each row represents one observation and each column represents one feature (including the intercept column of ones)
• $\boldsymbol{\beta}$ is a $(p+1) \times 1$ vector of coefficients: $\boldsymbol{\beta} = [\beta_0, \beta_1, \beta_2, \ldots, \beta_p]^T$

where:

• $n$ is the number of observations
• $p$ is the number of features (excluding the intercept)
• The superscript $T$ denotes the transpose operation

The predicted values can now be written as:

$$\hat{\mathbf{y}} = \mathbf{X}\boldsymbol{\beta}$$

where $\hat{\mathbf{y}} = [\hat{y}_1, \hat{y}_2, \ldots, \hat{y}_n]^T$ is the $n \times 1$ vector of predicted values.

This is much more compact than writing out each prediction individually. The sum of squared errors becomes:

$$\sum_{i=1}^n (y_i - \hat{y}_i)^2 = ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||_2^2$$

where $||\cdot||_2$ represents the L2 norm (Euclidean norm), which for a vector $\mathbf{v} = [v_1, v_2, \ldots, v_n]^T$ is defined as:

$$||\mathbf{v}||_2 = \sqrt{\sum_{i=1}^n v_i^2}$$

Therefore, $||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||_2^2 = \sum_{i=1}^n (y_i - \hat{y}_i)^2$ is the squared Euclidean distance between the actual values and our predictions.

Similarly, the penalty terms become:

• L1 penalty: $\lambda_1 ||\boldsymbol{\beta}||_1 = \lambda_1 \sum_{j=0}^p |\beta_j|$
• L2 penalty: $\lambda_2 ||\boldsymbol{\beta}||_2^2 = \lambda_2 \sum_{j=0}^p \beta_j^2$

where the summation includes all coefficients from $j=0$ (intercept) to $j=p$ (last feature).

Putting it all together, our Elastic Net objective function in matrix notation is:

$$\min_{\boldsymbol{\beta}} \left\{ ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||_2^2 + \lambda_1 ||\boldsymbol{\beta}||_1 + \lambda_2 ||\boldsymbol{\beta}||_2^2 \right\}$$

where:

• $\min_{\boldsymbol{\beta}}$ indicates we are finding the coefficient vector $\boldsymbol{\beta}$ that minimizes the objective function
• The objective function is the sum of three terms: data fit (SSE), L1 penalty, and L2 penalty

This notation makes it clear that we're minimizing a function of the coefficient vector $\boldsymbol{\beta}$, and it's much more convenient for mathematical analysis and computational implementation.

Understanding the mathematical properties of Elastic Net helps us predict how it will behave in different situations:

Sparsity and Grouping Effect: The L1 term encourages sparsity by driving some coefficients exactly to zero, while the L2 term encourages grouping of correlated features. This means that when features are highly correlated, Elastic Net tends to include the entire group rather than arbitrarily selecting one feature from the group (as LASSO might do).

Bias-Variance Tradeoff: As the regularization parameters $\lambda_1$ and $\lambda_2$ increase, the model becomes more biased (systematically wrong) but has lower variance (less sensitive to small changes in the data). This is the fundamental tradeoff in regularization.

Stability: The L2 component helps stabilize the solution when features are correlated, making the coefficient estimates more reliable and less sensitive to small changes in the data.

No Closed-Form Solution: Unlike Ridge regression, which has a closed-form solution, Elastic Net requires iterative optimization due to the non-differentiable L1 penalty. This makes it computationally more expensive but still more tractable than pure LASSO in many cases.

Convexity: The Elastic Net objective function is convex, which means it has a unique global minimum. This guarantees that our optimization algorithm will find the best solution.

One of the most insightful ways to understand how Elastic Net behaves is to examine the regularization path.how coefficients change as we vary the regularization strength. This visualization reveals the key difference between LASSO and Elastic Net when dealing with correlated features.

These regularization paths reveal the fundamental difference between LASSO and Elastic Net:

LASSO Path (Left): The red and blue lines (features 0 and 1, which are highly correlated) follow different trajectories. LASSO arbitrarily selects feature 0 and drives feature 1 to zero early in the path. This instability makes it difficult to interpret which feature is more important when features are correlated.

Elastic Net Path (Right): The red and blue lines stay close together throughout the regularization path. Both correlated features enter and exit the model together. This behavior is particularly useful when you have domain knowledge that certain features should be considered together.

The regularization path also shows how to select the optimal regularization strength: you would typically use cross-validation to find the α value that minimizes prediction error on held-out data, then read off the corresponding coefficients from these paths.

In practice, Elastic Net is often parameterized using a mixing parameter $\alpha$ and a total regularization strength $\lambda$. This alternative form makes it easier to understand the relationship between different regularization methods:

$$\min_{\boldsymbol{\beta}} \left\{ ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||_2^2 + \lambda \left( \alpha ||\boldsymbol{\beta}||_1 + \frac{1-\alpha}{2} ||\boldsymbol{\beta}||_2^2 \right) \right\}$$

where:

• $\alpha \in [0,1]$ is the mixing parameter (controls the balance between L1 and L2 penalties)
• $\lambda > 0$ is the total regularization strength (controls how much regularization we apply overall)

The relationship between the two parameterizations is:

$$\begin{aligned} \lambda_1 &= \lambda \alpha \\ \lambda_2 &= \lambda \frac{1-\alpha}{2} \end{aligned}$$

where:

• $\lambda_1$ is the L1 penalty strength from the original formulation
• $\lambda_2$ is the L2 penalty strength from the original formulation

This parameterization makes it much easier to understand the behavior:

• When $\alpha = 1$: We get pure LASSO (L1 only) because the L2 term becomes zero
• When $\alpha = 0$: We get pure Ridge (L2 only) because the L1 term becomes zero
• When $0 < \alpha < 1$: We get Elastic Net with a balance between L1 and L2 penalties

The factor of $\frac{1}{2}$ in the L2 term is included for mathematical convenience - it makes the derivatives cleaner and is a common convention in machine learning literature.

Scikit-learn uses a slightly different parameterization for computational efficiency:

$$\min_{\boldsymbol{\beta}} \left\{ \frac{1}{2n} ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||_2^2 + \alpha \left( \rho ||\boldsymbol{\beta}||_1 + \frac{1-\rho}{2} ||\boldsymbol{\beta}||_2^2 \right) \right\}$$

where:

• $n$ is the number of samples (observations)
• $\alpha > 0$ is the total regularization strength (equivalent to $\lambda$ in the alternative parameterization)
• $\rho \in [0,1]$ is the l1_ratio parameter (equivalent to $\alpha$ in the alternative parameterization)
• The factor $\frac{1}{2n}$ normalizes the data fit term by the sample size

In scikit-learn:

• alpha parameter controls the overall regularization strength ($\alpha$ in the formula above)
• l1_ratio parameter controls the mixing between L1 and L2 penalties ($\rho$ in the formula above)
• When l1_ratio = 1 ($\rho = 1$): Pure LASSO
• When l1_ratio = 0 ($\rho = 0$): Pure Ridge
• When 0 < l1_ratio < 1 ($0 < \rho < 1$): Elastic Net

• Sparsity and Grouping: The L1 term encourages sparsity (some coefficients exactly zero), while the L2 term encourages grouping of correlated features
• Bias-Variance Tradeoff: As $\lambda$ increases, bias increases but variance decreases
• Stability: The L2 term helps stabilize the solution when features are correlated
• No Closed-Form Solution: Like LASSO, Elastic Net requires iterative optimization due to the L1 penalty