Elastic Net Regularization › Implementation in Scikit-learn
Machine Learning from ScratchAll chapters Elastic Net Regularization › Implementation in Scikit-learnOne section · Sections in this chapter5 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We'll implement Elastic Net using scikit-learn to demonstrate how to apply this technique in practice. This tutorial walks through the complete workflow: data preparation, model training with automatic hyperparameter tuning, and evaluation. We'll use ElasticNetCV which automatically finds the best combination of regularization parameters through cross-validation.
Step 1: Import Libraries and Generate Data
First, we'll import the necessary libraries and create a synthetic dataset with correlated features to demonstrate Elastic Net's grouping effect.
import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
# Set random seed for reproducibility
np.random.seed(42)
# Generate synthetic data with correlated features
X, y = make_regression(
n_samples=1000, n_features=20, noise=50.0, random_state=42
)
# Create correlated features to demonstrate grouping effect
X[:, 1] = X[:, 0] + 0.3 * np.random.randn(1000) # Features 0 & 1 are correlated
X[:, 3] = 0.7 * X[:, 2] + 0.3 * np.random.randn(
1000
) # Features 2 & 3 are correlated
# Split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)Dataset: 1000 samples, 20 features Training set: 800 samples Test set: 200 samples
We've created a dataset with 1,000 samples and 20 features, where some features are intentionally correlated. Features 0 and 1 are highly correlated, as are features 2 and 3. This correlation structure will allow us to demonstrate Elastic Net's grouping effect.its ability to keep correlated features together in the model.
Step 2: Configure and Train the Model
Next, we'll set up an Elastic Net model with automatic hyperparameter tuning using ElasticNetCV. This approach tests multiple combinations of regularization parameters and selects the best one through cross-validation.
# Create Elastic Net with automatic hyperparameter selection
elastic_net = ElasticNetCV(
l1_ratio=[
0.1,
0.3,
0.5,
0.7,
0.9,
], # Mixing parameter: balance between L1 and L2
alphas=np.logspace(-3, 1, 20), # Regularization strength: from 0.001 to 10
cv=5, # 5-fold cross-validation for parameter selection
max_iter=2000, # Maximum iterations for convergence
random_state=42, # For reproducible results
n_jobs=-1, # Use all available CPU cores
)
# Create pipeline with standardization and Elastic Net
pipeline = Pipeline(
[
(
"scaler",
StandardScaler(),
), # Important: standardize features before regularization
("elastic_net", elastic_net),
]
)
# Fit the model on training data
pipeline.fit(X_train, y_train)The ElasticNetCV model will test 5 different l1_ratio values (controlling the L1/L2 balance) against 20 different alpha values (controlling overall regularization strength), resulting in 100 different parameter combinations. For each combination, it performs 5-fold cross-validation to estimate performance, then selects the best parameters.
The Pipeline ensures that feature standardization is applied consistently to both training and test data, which is important for Elastic Net since the regularization penalty is sensitive to feature scales.
Step 3: Evaluate Model Performance
Now let's evaluate how well the model performs on the test set and examine which hyperparameters were selected.
# Make predictions on test data
y_pred = pipeline.predict(X_test)
# Calculate performance metrics
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
# Extract the best Elastic Net model from the pipeline
best_elastic_net = pipeline.named_steps["elastic_net"]Performance Metrics: Mean Squared Error: 12234.5750 R² Score: 0.7076
The R² score indicates how much of the variance in the target variable our model explains. A value close to 1.0 suggests excellent predictive performance, while values closer to 0 indicate poor performance. The MSE provides the average squared difference between predictions and actual values.lower values indicate better fit.
Optimal Hyperparameters: Best l1_ratio: 0.900 Best alpha: 0.1274
The selected l1_ratio tells us the balance between L1 and L2 regularization that worked best for this dataset. A value closer to 1.0 means the model favored LASSO-like behavior (more sparsity), while values closer to 0 indicate Ridge-like behavior (keeping more features with small coefficients). The alpha value controls the overall strength of regularization.higher values mean more aggressive regularization.
Step 4: Examine Feature Selection
Let's look at which features the model selected and their coefficients to understand the sparsity pattern.
# Count and display non-zero coefficients
non_zero_coefs = 0
selected_features = []
for i, coef in enumerate(best_elastic_net.coef_):
if abs(coef) > 1e-6: # Non-zero coefficient threshold
non_zero_coefs += 1
selected_features.append((i, coef))Selected Features and Coefficients: Feature 0: 81.9982 Feature 1: 4.0983 Feature 2: 6.4044 Feature 4: 83.1459 Feature 5: -0.5909 Feature 6: 69.7527 Feature 7: 7.5425 Feature 8: -1.2363 Feature 9: 3.9866 Feature 10: 19.8315 Feature 11: 44.8573 Feature 12: 3.7041 Feature 13: 2.3340 Feature 14: -3.9652 Feature 15: 24.8593 Feature 16: -0.3934 Feature 17: 83.3584 Feature 18: 2.5290 Feature 19: 1.7641 Sparsity Summary: Features selected: 19/20 Sparsity ratio: 95.0%
The sparsity ratio shows what percentage of features the model retained. Elastic Net's automatic feature selection has eliminated features that contribute little to predictions, simplifying the model and potentially improving interpretability. Notice how correlated features (like 0 and 1, or 2 and 3) tend to be selected or eliminated together. This is the grouping effect in action.
Alternative: Manual Hyperparameter Tuning with GridSearchCV
While ElasticNetCV is recommended for most cases, you can also use GridSearchCV if you need more control over the search process or want to tune additional parameters.
from sklearn.linear_model import ElasticNet
from sklearn.model_selection import GridSearchCV
# Define parameter grid for manual search
param_grid = {
"elastic_net__l1_ratio": [0.1, 0.5, 0.9], # Mixing parameter values
"elastic_net__alpha": [0.01, 0.1, 1.0], # Regularization strength values
}
# Create pipeline with manual Elastic Net
manual_pipeline = Pipeline(
[
("scaler", StandardScaler()),
("elastic_net", ElasticNet(max_iter=2000, random_state=42)),
]
)
# Perform grid search with cross-validation
grid_search = GridSearchCV(
manual_pipeline,
param_grid,
cv=3, # 3-fold cross-validation
scoring="neg_mean_squared_error", # Minimize MSE
n_jobs=-1, # Use all CPU cores
)
# Fit the grid search
grid_search.fit(X_train, y_train)GridSearchCV Results:
Best parameters: {'elastic_net__alpha': 0.1, 'elastic_net__l1_ratio': 0.9}
Best cross-validation MSE: 12067.6621
Comparison with ElasticNetCV:
ElasticNetCV l1_ratio: 0.900
ElasticNetCV alpha: 0.1274
GridSearchCV l1_ratio: 0.9
GridSearchCV alpha: 0.1Both approaches should yield similar results, though ElasticNetCV typically explores a finer grid of parameters and is optimized specifically for Elastic Net. The GridSearchCV approach offers more flexibility if you want to tune additional hyperparameters or use custom scoring functions.
Elastic Net is highly sensitive to feature scales. Like Ridge regression, Elastic Net requires all features to be on the same scale. Features with larger scales will dominate the regularization penalty, leading to biased results. Use StandardScaler or MinMaxScaler before applying Elastic Net to ensure fair treatment of all features.
Key Parameters
Below are the main parameters that affect how Elastic Net works and performs.
-
alpha: Overall regularization strength (default: 1.0). Higher values apply stronger regularization, leading to smaller coefficients and more sparsity. Start with values between 0.01 and 10, using cross-validation to find the optimal value for your dataset. -
l1_ratio: Mixing parameter controlling the balance between L1 and L2 penalties (default: 0.5). Values range from 0 (pure Ridge) to 1 (pure LASSO). Use 0.5 for a balanced approach, or tune this parameter if you know whether you need more sparsity (higher values) or more grouping (lower values). -
max_iter: Maximum number of iterations for the optimization algorithm (default: 1000). Increase to 2000 or higher if you encounter convergence warnings, especially with large datasets or strong regularization. -
tol: Tolerance for optimization convergence (default: 1e-4). Smaller values lead to more precise solutions but longer training times. The default works well for most applications. -
random_state: Seed for reproducibility (default: None). Set to an integer to ensure consistent results across runs, especially important when comparing different models. -
selection: Method for coefficient updates during coordinate descent (default: 'cyclic'). Options are 'cyclic' (updates coefficients in order) or 'random' (updates in random order). Random selection can be faster for large datasets.
Key Methods
The following are the most commonly used methods for interacting with Elastic Net models.
-
fit(X, y): Trains the Elastic Net model on the training data X and target values y. Performs coordinate descent optimization to find optimal coefficients. -
predict(X): Returns predicted values for input data X using the learned coefficients. -
score(X, y): Returns the R² score (coefficient of determination) on the given test data. Values closer to 1.0 indicate better model performance. -
get_params(): Returns a dictionary of all model parameters. Useful for inspecting the current configuration. -
set_params(**params): Sets model parameters. Useful for updating parameters without creating a new model instance.