Polynomial Regression › Implementation in Scikit-learn
Machine Learning from ScratchAll chapters Polynomial Regression › Implementation in Scikit-learnOne section · Sections in this chapter5 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Scikit-learn provides excellent tools for implementing polynomial regression through the PolynomialFeatures transformer and LinearRegression estimator. Let's walk through a complete implementation that demonstrates how to fit and evaluate polynomial regression models.
Setting Up the Data
We'll start by generating synthetic data with a known cubic relationship, which will allow us to see how well polynomial regression can recover the true underlying function.
import numpy as np
from sklearn.model_selection import train_test_split
# Generate synthetic data with cubic relationship
np.random.seed(42)
n_samples = 100
x = np.linspace(-2, 2, n_samples)
y_true = 2 * x**3 - 3 * x**2 + x + 1
noise = np.random.normal(0, 0.5, n_samples)
y = y_true + noise
# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
x.reshape(-1, 1), y, test_size=0.3, random_state=42
)The data follows a cubic polynomial with added Gaussian noise. By splitting into training (70%) and test (30%) sets, we can evaluate how well the model generalizes to unseen data.
Building a Polynomial Regression Pipeline
The most efficient way to implement polynomial regression in scikit-learn is using a Pipeline that combines the PolynomialFeatures transformer with LinearRegression. This ensures that the same polynomial transformation is consistently applied to both training and test data.
from sklearn.metrics import mean_squared_error
# Create pipeline with degree-3 polynomial features
poly_reg = Pipeline(
[
("poly", PolynomialFeatures(degree=3, include_bias=True)),
("linear", LinearRegression()),
]
)
# Train the model
poly_reg.fit(X_train, y_train)
# Make predictions
y_pred = poly_reg.predict(X_test)
# Calculate performance metrics
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)Model Performance: Mean Squared Error: 0.1395 R² Score: 0.9980
The low MSE and high R² score (close to 1.0) indicate excellent model performance. Since our data was generated from a cubic function and we used a degree-3 polynomial, the model successfully captures the true underlying relationship. The R² score tells us that the model explains nearly all the variance in the test data.
Model Coefficients: Intercept: 0.8867 Coefficients: [ 0. 1.2487505 -2.96269465 1.90648781]
The coefficients show the contribution of each polynomial term. Comparing these to our true function (), we can see the model has successfully recovered values close to the true parameters, despite the added noise.
Comparing Different Polynomial Degrees
A critical question in polynomial regression is choosing the right degree. Let's compare models with different degrees to understand the bias-variance trade-off.
# Compare polynomial degrees from 1 to 5
degrees = [1, 2, 3, 4, 5]
results = []
for degree in degrees:
# Create and fit model
model = Pipeline(
[
("poly", PolynomialFeatures(degree=degree, include_bias=True)),
("linear", LinearRegression()),
]
)
model.fit(X_train, y_train)
y_pred_deg = model.predict(X_test)
# Calculate metrics
mse_deg = mean_squared_error(y_test, y_pred_deg)
r2_deg = r2_score(y_test, y_pred_deg)
n_features = model.named_steps["poly"].n_output_features_
results.append(
{
"degree": degree,
"mse": mse_deg,
"r2": r2_deg,
"n_features": n_features,
}
)Comparison of Polynomial Degrees: Degree MSE R² Features --------------------------------------------- 1 21.7172 0.6895 2 2 6.3839 0.9087 3 3 0.1395 0.9980 4 4 0.1366 0.9980 5 5 0.1388 0.9980 6 Best degree: 4 (MSE = 0.1366)
The comparison reveals important insights about model selection:
- Degree 1 (Linear): High MSE and low R² indicate underfitting. A straight line cannot capture the cubic relationship.
- Degree 2 (Quadratic): Improved performance but still underfits the true cubic function.
- Degree 3 (Cubic): Optimal performance with the lowest MSE. This matches the true data-generating process.
- Degrees 4-5: Similar or slightly worse performance than degree 3, suggesting potential overfitting to noise in the training data.
The degree-3 model achieves the best balance between bias and variance, as expected since the true relationship is cubic. Higher degrees don't improve performance and risk overfitting, especially with limited training data.
Visualizing Model Predictions
Let's visualize how different polynomial degrees fit the data to better understand their behavior.
The visualizations clearly show the progression from underfitting (degree 1) to optimal fit (degree 3) to potential overfitting (degree 5). The degree-3 model smoothly captures the underlying cubic trend, while the degree-5 model shows slight oscillations that fit noise rather than the true pattern.
Key Parameters
Below are the main parameters that control polynomial regression behavior and performance.
PolynomialFeatures Parameters:
degree: The maximum degree of polynomial features to generate (default: 2). This is the most critical parameter. Start with low values (2-3) and increase only if needed. Higher degrees capture more complex patterns but risk overfitting.include_bias: Whether to include the intercept column of ones (default: True). Typically set toTruewhen using withLinearRegression, which expects the bias term in the feature matrix.interaction_only: IfTrue, only interaction features are produced, not powers (default: False). Use this when you want to model interactions between variables without including squared or higher-power terms.
LinearRegression Parameters:
fit_intercept: Whether to calculate the intercept (default: True). When usingPolynomialFeatureswithinclude_bias=True, this is typically set toFalseto avoid redundancy, though scikit-learn handles this automatically.copy_X: Whether to copy the input data (default: True). Set toFalseto save memory if you don't need the original data preserved.
Pipeline Benefits:
- Automatically applies the same polynomial transformation to training and test data
- Simplifies cross-validation and hyperparameter tuning
- Ensures reproducible preprocessing steps
- Makes code more maintainable and less error-prone
Key Methods
The following methods are used to train and apply polynomial regression models.
fit(X, y): Trains the polynomial regression model on input features X and target values y. This applies the polynomial transformation and fits the linear regression coefficients.predict(X): Returns predicted values for input data X. The polynomial transformation is automatically applied before making predictions.score(X, y): Returns the R² score (coefficient of determination) on the given test data. Values closer to 1.0 indicate better fit.get_params(): Returns the parameters of the pipeline components. Useful for inspecting or modifying the model configuration.set_params(**params): Sets the parameters of the pipeline. Commonly used with grid search for hyperparameter tuning.