Multicollinearity › The Impact on Coefficient Stability

Michael BrenndoerferApril 13, 20254 min read
Machine Learning from ScratchAll chapters
Part 3 of 9Foundations
Chapter 28Multicollinearity › The Impact on Coefficient Stability
Reading section 4 of 13 · The Impact on Coefficient Stability · 4 minSection 4 of 13 · 4 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 28 of 66

Section 4 of 13 · 4 min read

Multicollinearity › The Impact on Coefficient StabilityOne section · Sections in this chapter4 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One of multicollinearity's primary concerns is its effect on coefficient stability. When variables are highly correlated, small changes in the data can cause substantial changes in coefficient estimates, making them less reliable for interpretation. This instability occurs because the regression model has difficulty distinguishing between the individual contributions of correlated variables.

To understand this concept, we can use bootstrap sampling to show how coefficient estimates vary under different multicollinearity conditions. Bootstrap sampling involves repeatedly resampling our data and fitting the model to see how much the coefficients change.

Let's compare two scenarios: one with independent variables (low multicollinearity) and another with highly correlated variables (high multicollinearity):

Out[6]:
Visualization
Scatter plot of bootstrap coefficient estimates under low multicollinearity conditions.
Coefficient stability under low multicollinearity conditions. Each point represents a bootstrap sample coefficient estimate, with the horizontal line showing the mean. When variables are independent, coefficient estimates are stable and tightly clustered around their true values.
Scatter plot showing coefficient instability under high multicollinearity with wide dispersion of estimates for age, height, weight, and BMI variables.
Coefficient instability under high multicollinearity conditions. The wide scatter of bootstrap coefficient estimates demonstrates how multicollinearity leads to unstable and unreliable coefficient estimates. This variability makes it difficult to determine the true individual effect of each predictor variable.
Out[7]:
Console
Low Multicollinearity Scenario:
Correlation Matrix:
[[ 1.         -0.12962222  0.18560173]
 [-0.12962222  1.         -0.0342348 ]
 [ 0.18560173 -0.0342348   1.        ]]

VIF values: 
[np.float64(11.400090793884045), np.float64(6.848636461932772), np.float64(17.179461809888984)]
Out[8]:
Console
High Multicollinearity Scenario:
Correlation Matrix:
[[ 1.         -0.2722509  -0.02179877  0.27471253]
 [-0.2722509   1.          0.23289708 -0.9872858 ]
 [-0.02179877  0.23289708  1.         -0.10582219]
 [ 0.27471253 -0.9872858  -0.10582219  1.        ]]

VIF values: 
[np.float64(10.11406861947651), np.float64(7132.583008656807), np.float64(15382.274755070397), np.float64(1674.8187604095335)]

Coefficient Stability Analysis:
Low multicollinearity - Max coefficient std: 0.364
High multicollinearity - Max coefficient std: 1.222
Instability ratio: 3.4x more variable under high multicollinearity