Lasso Regularization (L1 Regularization) › Formula

LASSO regularization is formulated as a minimization optimization problem. The goal is to find the set of coefficients that fit the data well (by minimizing the sum of squared errors) while keeping the model simple by penalizing large coefficients. In other words, LASSO seeks to minimize a loss function that combines the usual error term with an additional penalty based on the absolute values of the coefficients. The formula for this optimization problem is:

$$\text{SSE} + \lambda \sum_{j=1}^p |\beta_j|$$

where:

• $\lambda$: regularization parameter (controls the strength of the penalty)
• $p$: number of features in the model
• $\beta_j$: coefficient for the $j$-th feature (where $j = 1, 2, \ldots, p$)
• $|\beta_j|$: absolute value of the $j$-th coefficient (ensures non-negative penalty)
• $\text{SSE}$: sum of squared errors between predicted and actual values

The regularization parameter $\lambda$ controls the strength of the regularization. A larger $\lambda$ results in more shrinkage of the coefficients, and a smaller $\lambda$ results in less shrinkage.

Written as a minimization problem, it is:

$$\min_{\beta} \left\{ \text{SSE} + \lambda \sum_{j=1}^p |\beta_j| \right\}$$

This is read as: "minimize the sum of squared errors plus the regularization penalty, with respect to the coefficients". The fully written-out formula is:

$$\min_{\beta} \left\{ \sum_{i=1}^n \left( y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij} \right)^2 + \lambda \sum_{j=1}^p |\beta_j| \right\}$$

where:

• $n$: number of observations in the dataset
• $y_i$: actual target value for observation $i$
• $\beta_0$: intercept term (typically not penalized)
• $x_{ij}$: value of feature $j$ for observation $i$
• $\lambda$: regularization parameter
• $p$: number of features

The left side of the equations, the sum of squared errors (SSE), should look familiar from the previous section on ordinary least squares (OLS). The right side is the regularization penalty. However, unlike OLS, LASSO requires us to minimize the SSE plus the regularization penalty. OLS allows us to directly compute the optimal coefficients using a closed-form solution and vector algebra, but LASSO does not have this property.

This means we need to use optimization techniques to find the best coefficients, rather than relying on a simple formula. In other words, we are using iterative optimization methods to find the coefficients that minimize the combined objective function.