Multicollinearity › Model Selection Under Multicollinearity: R-squared vs Adjusted R-squared
Machine Learning from ScratchAll chapters Multicollinearity › Model Selection Under Multicollinearity: R-squared vs Adjusted R-squaredOne section · Sections in this chapter8 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
An important distinction emerges between R-squared and adjusted R-squared in the presence of multicollinearity. R-squared represents the proportion of variance in the outcome variable explained by the model, but multicollinearity can artificially inflate this measure, creating a misleading impression of model quality.
Adjusted R-squared provides a more realistic assessment by accounting for the number of predictors in the model, imposing a penalty for adding variables that don't contribute unique information. When redundant predictors are included, adjusted R-squared decreases, signaling that these additional variables don't improve the model's explanatory power.
Let's examine how multicollinearity affects these model selection metrics:
Multicollinearity Impact on Model Metrics: ============================================================ Correlation R² Adj R² Difference VIF ------------------------------------------------------------ 0.1 0.534 0.504 0.030 15.1 0.3 0.403 0.364 0.039 19.9 0.5 0.513 0.482 0.031 14.7 0.7 0.597 0.571 0.026 17.5 0.9 0.483 0.450 0.033 1.3
Other Remedial Actions
There are several strategies, beyond Ridge regression, that can be used to address multicollinearity in regression models:
Variable Selection Methods
One approach is to use variable selection techniques, such as stepwise regression, which systematically add or remove predictors based on statistical criteria to identify the most informative subset of variables. Another method is Principal Component Regression (PCR), which replaces the original, correlated variables with a smaller set of uncorrelated principal components. Partial Least Squares (PLS) is a related technique that, like PCR, transforms the predictors but does so in a way that maximizes their ability to predict the outcome.
Data Transformation
Transforming the data can also help mitigate multicollinearity. Centering and scaling the variables (standardizing them to have mean zero and unit variance) makes them more comparable and can sometimes reduce collinearity. Highly correlated variables can be combined into a single composite variable, reducing redundancy. Additionally, domain knowledge can be useful for selecting the most relevant predictors.
Regularization Techniques
Regularization methods add penalties to the regression coefficients to prevent them from becoming too large. Ridge regression, which uses an L2 penalty, shrinks all coefficients toward zero but does not set any exactly to zero. Lasso regression, with an L1 penalty, can shrink some coefficients all the way to zero, effectively performing variable selection. Elastic Net combines both L1 and L2 penalties, balancing the strengths of Ridge and Lasso.
Advanced Methods
More advanced approaches include Bayesian regression, which incorporates prior beliefs or expert knowledge about the likely values of coefficients, and robust regression techniques, which are less sensitive to the effects of multicollinearity and other violations of standard regression assumptions.