Multicollinearity › Solutions to Multicollinearity: Ridge Regression

Michael BrenndoerferApril 13, 20257 min read
Machine Learning from ScratchAll chapters
Part 3 of 9Foundations
Chapter 28Multicollinearity › Solutions to Multicollinearity: Ridge Regression
Reading section 7 of 13 · Solutions to Multicollinearity: Ridge Regression · 7 minSection 7 of 13 · 7 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 28 of 66

Section 7 of 13 · 7 min read

Multicollinearity › Solutions to Multicollinearity: Ridge RegressionOne section · Sections in this chapter7 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When multicollinearity is detected, several remedial approaches are available. Ridge regression is one of the most effective solutions, as it handles multicollinearity automatically through regularization.

Ridge regression adds a penalty term to the regression equation that shrinks coefficients toward zero, reducing their variance and making them more stable. The regularization parameter α controls the amount of shrinkage.higher values lead to more shrinkage and greater stability.

The key advantage of Ridge regression is that it doesn't require removing variables; instead, it stabilizes all coefficients while keeping them in the model. This makes it particularly useful when all variables are theoretically important.

Let's see how Ridge regression handles our multicollinear health data:

Out[9]:
Visualization
Line plot showing ridge regression coefficient paths for age, height, weight, and BMI variables as alpha increases on a logarithmic scale.
Ridge regression coefficient paths showing regularization shrinks coefficients as the penalty parameter α increases. This visualization demonstrates how ridge regression handles multicollinearity by stabilizing coefficient estimates and reducing their variance.
Out[10]:
Visualization
Bar chart displaying ordinary least squares coefficient estimates for age, height, weight, and BMI under multicollinearity conditions.
Ordinary Least Squares (OLS) coefficient estimates showing unstable coefficients under multicollinearity. The bar chart displays the magnitude of each coefficient, with value labels above each bar. These coefficients are unstable due to the high correlation between predictor variables.
Bar chart showing ridge regression coefficient estimates with optimal alpha regularization.
Ridge regression coefficient estimates showing stable coefficients through regularization. The bar chart demonstrates how Ridge regression gives more stable coefficient estimates by applying shrinkage, reducing the impact of multicollinearity and showing significant coefficient stabilization.
Out[11]:
Console
Coefficient Comparison:
==================================================
Variable OLS        Ridge      Difference  
--------------------------------------------------
Age      -4.752     -4.707     0.045       
Height   -9.009     -5.938     3.071       
Weight   0.344      0.256      0.088       
BMI      -9.546     -6.499     3.047       

Optimal Ridge α: 0.206

Ridge Regression Analysis:
OLS coefficient range: 9.890
Ridge coefficient range: 6.755
Shrinkage ratio: 0.683 (lower = more stable)
Coefficient stabilization: 31.7% reduction in range

Example: Condition Index Analysis

This example shows how to use condition index and eigenvalues to diagnose multicollinearity:

Out[12]:
Visualization
Line plot of eigenvalues in descending order with multicollinearity threshold line, showing which principal components have eigenvalues below 0.1 indicating multicollinearity.
Eigenvalue analysis showing the magnitude of eigenvalues in descending order. Small eigenvalues (below the red dashed threshold line) indicate multicollinearity, as they represent directions in the data space with little variation. This plot helps identify how many principal components are affected by multicollinearity and the severity of the problem.
Bar chart of condition indices for each principal component, with color coding showing high multicollinearity above 30 and moderate above 10.
Condition index analysis displaying multicollinearity severity for each principal component. Values above 30 (red bars) indicate high multicollinearity, while values between 10-30 (orange bars) suggest moderate multicollinearity. This quantitative assessment complements the eigenvalue analysis by providing specific thresholds for multicollinearity diagnosis.
Out[13]:
Console
Multicollinearity Diagnostics:
========================================
Largest eigenvalue: 2.043
Smallest eigenvalue: 0.005
Condition index: 20.3
Number of eigenvalues < 0.1: 1
Number of condition indices > 30: 0