Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Elastic Net Regularization › Visualizing Elastic Net

Michael BrenndoerferJune 12, 20257 min read

Machine Learning from ScratchAll chapters

Part 4 of 9Regression Models

Chapter 36Elastic Net Regularization › Visualizing Elastic Net

Reading section 3 of 8 · Visualizing Elastic Net · 7 minSection 3 of 8 · 7 min

Section 3 of 8 · 7 min read

Elastic Net Regularization › Visualizing Elastic NetOne section · Sections in this chapter3 / 8 ↓

Sections in this chapter8

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's visualize how Elastic Net behaves by comparing it with LASSO and Ridge across different parameter values.

Out[5]:

Visualization

Ridge regression coefficients with λ=0.01 showing slight shrinkage. — Ridge regression with λ=0.01: All coefficients remain non-zero but are slightly shrunk from OLS estimates. Notice how correlated features 0 and 1 maintain similar values.

Ridge regression coefficients with λ=0.1 showing moderate shrinkage. — Ridge regression with λ=0.1: Coefficients are further shrunk toward zero while maintaining the grouping effect for correlated features.

Ridge regression coefficients with λ=1.0 showing strong shrinkage. — Ridge regression with λ=1.0: Strong shrinkage brings all coefficients closer to zero, but no feature selection occurs.

Out[6]:

Visualization

LASSO coefficients with α=0.01 showing minimal feature selection. — LASSO with α=0.01: Minimal regularization, most features retained with coefficients close to OLS estimates.

LASSO coefficients with α=0.1 showing moderate feature selection. — LASSO with α=0.1: Moderate feature selection begins, some coefficients driven to zero while others remain.

LASSO coefficients with α=1.0 showing strong feature selection. — LASSO with α=1.0: Strong feature selection, only the most important features retained. Note potential instability with correlated features.

Out[7]:

Visualization

Elastic Net coefficients with α=0.1 showing mostly Ridge behavior. — Elastic Net with α=0.1 (mostly Ridge): Behaves similarly to Ridge, keeping most features with small coefficients.

Elastic Net coefficients with α=0.5 showing balanced behavior. — Elastic Net with α=0.5 (balanced): Combines feature selection with stability.

Elastic Net coefficients with α=0.9 showing mostly LASSO behavior. — Elastic Net with α=0.9 (mostly LASSO): Strong feature selection while maintaining some stability for correlated features.

Row 1 - Ridge Regression (λ = 0.01, 0.1, 1.0): As the regularization parameter λ increases, all coefficients shrink toward zero but remain non-zero. Notice how correlated features 0 and 1 maintain similar coefficient values across all λ values. The coefficients become progressively smaller as λ increases, but no feature is ever eliminated.

Row 2 - LASSO (α = 0.01, 0.1, 1.0): As the regularization parameter α increases, LASSO performs automatic feature selection by driving some coefficients exactly to zero. Notice how LASSO might arbitrarily select one feature from the correlated pair (features 0 and 1) when α becomes large. The sparsity increases dramatically as α increases, with only the most important features remaining.

Row 3 - Elastic Net (α = 0.1, 0.5, 0.9): This row shows the mixing parameter α (l1_ratio) controlling the balance between L1 and L2 penalties. When α = 0.1, the model behaves more like Ridge, keeping most features with small coefficients. As α increases to 0.5 and 0.9, the L1 component becomes stronger, leading to more sparsity while still maintaining some stability for correlated features. Notice how features 0 and 1 tend to be kept together more consistently than in pure LASSO.

Out[8]:

Visualization

Bar chart comparing coefficient estimates from Ridge, LASSO, and Elastic Net regularization methods. — Side-by-side comparison of coefficient estimates across Ridge, LASSO, and Elastic Net methods with moderate regularization. Ridge (coral) keeps all features with moderate coefficients. LASSO (blue) performs aggressive feature selection, setting some coefficients exactly to zero, but may arbitrarily choose between correlated features. Elastic Net (teal) gives a balanced approach, maintaining some sparsity while preserving the grouping effect for correlated features.

The side-by-side comparison reveals the basic trade-offs between these methods. Ridge (red bars) keeps all features with moderate coefficients. LASSO (blue bars) performs aggressive feature selection, setting some coefficients exactly to zero, but may arbitrarily choose between correlated features. Elastic Net (green bars) gives a balanced approach, maintaining some sparsity while preserving the grouping effect for correlated features. This makes Elastic Net particularly useful when you have correlated features and want both interpretability and stability.