Multicollinearity › Lasso Regression: Automatic Variable Selection

Michael BrenndoerferApril 13, 20255 min read
Machine Learning from ScratchAll chapters
Part 3 of 9Foundations
Chapter 28Multicollinearity › Lasso Regression: Automatic Variable Selection
Reading section 10 of 13 · Lasso Regression: Automatic Variable Selection · 5 minSection 10 of 13 · 5 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 28 of 66

Section 10 of 13 · 5 min read

Multicollinearity › Lasso Regression: Automatic Variable SelectionOne section · Sections in this chapter10 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Lasso regression provides a different approach to handling multicollinearity by performing automatic variable selection. Unlike Ridge regression, which shrinks all coefficients but keeps all variables, Lasso can set some coefficients exactly to zero, effectively removing variables from the model.

This makes Lasso particularly useful when:

  • We suspect some variables are redundant
  • We want to identify the most important predictors
  • We need a simpler, more interpretable model

Lasso uses an L1 penalty that can shrink coefficients to exactly zero, performing both regularization and variable selection simultaneously. The regularization parameter α controls the amount of shrinkage.higher values lead to more variables being removed.

Let's see how Lasso regression performs variable selection on our multicollinear health data:

Out[19]:
Visualization
Lasso regression coefficient paths showing how L1 regularization shrinks coefficients to exactly zero for less important variables like height, weight, and BMI.
Lasso regression coefficient paths showing how the L1 penalty shrinks coefficients to zero as the regularization parameter α increases. This visualization demonstrates Lasso's ability to perform automatic variable selection by setting coefficients of less important variables to exactly zero.
Grouped bar chart comparing coefficient estimates across OLS, Ridge, and Lasso methods, showing how Lasso performs automatic variable selection by setting some coefficients to zero.
Comparison of variable selection between different regularization methods. The grouped bars compare OLS, Ridge, and Lasso coefficient estimates for every predictor, illustrating how Lasso provides both variable selection and multicollinearity handling in a single framework.
Out[20]:
Console
Model Comparison:
================================================================================
Method     R²       MSE      Selected Variables   Non-zero Coefs 
--------------------------------------------------------------------------------
OLS        0.456    25.985   Age, Height, Weight, BMI, Systolic_BP, Exercise 6              
Ridge      0.455    26.015   Age, Height, Weight, BMI, Systolic_BP, Exercise 6              
Lasso      0.449    26.321   Age, Exercise        2              

Optimal Lasso α: 0.2947
Lasso selected: Age, Exercise
Variables removed by Lasso: Height, Weight, BMI, Systolic_BP

VIF Analysis (Original Variables):
Age: 1.4 (Low)
Height: 103.4 (High)
Weight: 3.2 (Low)
BMI: 99.5 (High)
Systolic_BP: 1.3 (Low)
Exercise: 1.1 (Low)