Lasso Regularization (L1 Regularization) › Choosing

Michael BrenndoerferApril 19, 20257 min read
Machine Learning from ScratchAll chapters
Part 4 of 9Regression Models
Chapter 35Lasso Regularization (L1 Regularization) › Choosing
Reading section 7 of 11 · Choosing · 7 minSection 7 of 11 · 7 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 35 of 66

Section 7 of 11 · 7 min read

Lasso Regularization (L1 Regularization) › ChoosingOne section · Sections in this chapter7 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The regularization parameter λ\lambda is a hyperparameter, which is a value that is not learned from the data during model training but is set before training begins. The choice of λ\lambda is important:

  • If λ\lambda is too small, the penalty is weak and the model may overfit (behave like ordinary least squares).
  • If λ\lambda is too large, the penalty is too strong and the model may underfit (set too many coefficients to zero, losing important information).

To find the best λ\lambda, we typically use a process called hyperparameter search. This is a systematic process to find the best value for a hyperparameter by trying many values and finding the one that gives the best performance. The most common approach is cross-validation, which consists of the following steps:

  1. Define a range of candidate λ\lambda values (for example, from very small to quite large), e.g. [0.01, 0.05, 0.1, 0.5, 1.0, 2.0]
  2. For each λ\lambda value:
    • Train the LASSO model on a subset of the data (the training set).
    • Evaluate its performance on a separate subset (the validation set), using a metric like mean squared error (MSE).
  3. Select the λ\lambda that gives the best validation performance (i.e., the lowest error).

This process is often automated using tools like GridSearchCV in scikit-learn, which systematically tries many values and finds the optimal one.

Visualizing the Cross-Validation Process

To better understand how cross-validation helps us choose the optimal λ\lambda, let's visualize how model performance changes across different regularization strengths. The following plot demonstrates the bias-variance tradeoff that occurs as we vary λ\lambda.

Out[4]:
Visualization
Cross-validation error curves showing U-shaped validation error with optimal regularization strength marked by vertical line, balancing bias-variance tradeoff.
Cross-validation error curves showing the bias-variance tradeoff in LASSO regularization. The blue line shows training error increasing with regularization strength, while the red line shows validation error forming a U-shape. The optimal λ (green vertical line) minimizes validation error, balancing model complexity and generalization. The shaded red region represents standard error across cross-validation folds. Three labeled zones identify overfitting (λ too small), optimal (best generalization), and underfitting (λ too large) regions.
Feature selection plot showing selected features decreasing from 10 toward zero as regularization increases; the cross-validation optimum retains 8 features and a horizontal reference marks the true count of 3.
Feature selection behavior across regularization strengths. As λ increases from left to right, LASSO progressively eliminates features, reducing the model from 10 features to nearly zero. At the error-minimizing λ (green vertical line), LASSO retains 8 features; stronger regularization later reaches the true 3-feature level (gray horizontal line) before eliminating additional features.

This visualization reveals several important insights about choosing λ\lambda:

  1. Training Error (Blue Line): Increases monotonically as λ\lambda increases. This is expected because stronger regularization constrains the model more, preventing it from fitting the training data as closely.

  2. Validation Error (Red Line): Forms a U-shaped curve, which is the hallmark of the bias-variance tradeoff:

    • Left side (small λ\lambda): The model overfits the training data, leading to poor generalization and high validation error.
    • Bottom (optimal λ\lambda): The sweet spot where the model balances fitting the training data and generalizing to new data.
    • Right side (large λ\lambda): The model underfits, with too much regularization preventing it from capturing important patterns.

  3. Feature Selection (Bottom Panel): Shows how the number of selected features decreases as regularization increases. The error-minimizing λ\lambda retains 8 features in this cross-validation run; with stronger regularization, the path later reaches the 3 truly important features marked by the horizontal dashed line.

  4. Standard Error Bands: The shaded region around the validation error shows the variability across different cross-validation folds, helping us assess the stability of our performance estimates.

The optimal λ\lambda (marked by the green vertical line) minimizes the validation error. This is why cross-validation is the standard approach for hyperparameter selection in LASSO and other regularized models.