Lasso Regularization (L1 Regularization) › Summary

LASSO (L1) regularization is a technique for building simpler, more interpretable regression models by encouraging sparsity.shrinking some coefficients exactly to zero and thus performing feature selection. The core idea is captured in the summarized form of the LASSO objective, which adds a penalty proportional to the sum of the absolute values of the coefficients, controlled by the regularization parameter $\lambda$.

To make this precise and computationally tractable, we use the matrix form of the objective, as shown above:

$$\min_{\beta} \left\{ \frac{1}{2n} \|y - X\beta\|_2^2 + \alpha \|\beta\|_1 \right\}$$

where:

• $\beta$: $(p+1) \times 1$ vector of coefficients to be optimized (including intercept $\beta_0$)
• $n$: number of observations in the dataset
• $y$: $n \times 1$ target vector containing actual values
• $X$: $n \times (p+1)$ feature matrix (includes column of ones for intercept)
• $\alpha$: regularization parameter in scikit-learn (controls penalty strength)
• $\|y - X\beta\|_2^2$: squared L2 (Euclidean) norm of residuals, equal to $\sum_{i=1}^n (y_i - \hat{y}_i)^2$
• $\|\beta\|_1$: L1 (Manhattan) norm of coefficients, equal to $\sum_{j=1}^p |\beta_j|$ (intercept not penalized)

This formulation enables efficient computation and connects LASSO to broader optimization theory.

However, unlike ordinary least squares, LASSO does not have a closed-form solution because the L1 penalty makes the problem non-differentiable at zero. This is what allows LASSO to set coefficients exactly to zero, a property that distinguishes it from Ridge (L2) regularization (which we will cover in the next section), which typically only shrinks coefficients but rarely eliminates them.

To solve the LASSO problem, we need iterative optimization algorithms. One such method is the Iterative Shrinkage-Thresholding Algorithm (ISTA), described above. ISTA separates the smooth (least squares) and non-smooth (L1 penalty) parts of the objective, alternating between a gradient step and a soft-thresholding step. This practical algorithm is how we compute the LASSO solution in practice.

In summary: the summarized form tells us the goal, the matrix form gives us a precise and efficient way to express it, and ISTA (or similar algorithms) provides the means to find the solution. Understanding the distinction and relationship between these is key: the forms define what LASSO is optimizing, while ISTA is how we solve it. Choosing the right value of $\lambda$ or $\alpha$ (often via cross-validation) is important for balancing model simplicity and predictive performance.