Logistic Regression › Implementation
Machine Learning from ScratchAll chapters Logistic Regression › ImplementationOne section · Sections in this chapter5 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
This section demonstrates how to implement logistic regression using scikit-learn, from preprocessing through evaluation and interpretation of the results.
Basic Implementation
We'll start by creating a synthetic dataset that simulates student exam outcomes based on study hours and previous test scores. This example demonstrates the complete workflow from data preparation through model evaluation.
import numpy as np
from sklearn.model_selection import train_test_split
# Create sample data
np.random.seed(42)
n_samples = 1000
# Generate features
study_hours = np.random.normal(5, 2, n_samples)
study_hours = np.clip(study_hours, 0, 10) # Clip to realistic range
previous_scores = np.random.normal(75, 15, n_samples)
previous_scores = np.clip(previous_scores, 0, 100) # Clip to realistic range
# Create target based on realistic relationship
# Higher study hours and scores increase probability of passing
log_odds = -8 + 0.5 * study_hours + 0.05 * previous_scores
prob_pass = 1 / (1 + np.exp(-log_odds))
y = np.random.binomial(1, prob_pass, n_samples)
# Combine features
X = np.column_stack([study_hours, previous_scores])
# Split data with stratification to maintain class balance
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)Now we'll create a pipeline that combines feature scaling with logistic regression. Using a pipeline ensures that preprocessing steps are applied consistently during both training and prediction.
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
# Create pipeline with scaling and logistic regression
pipeline = make_pipeline(
StandardScaler(), LogisticRegression(random_state=42, max_iter=1000)
)
# Train the model
pipeline.fit(X_train, y_train)
# Make predictions
y_pred = pipeline.predict(X_test)
y_pred_proba = pipeline.predict_proba(X_test)[:, 1]Model Performance
Let's evaluate the model's performance from several metrics, each of which captures a different aspect of how well it performs.
Model Performance: Accuracy: 0.800 ROC AUC: 0.762
The accuracy of approximately 0.81 indicates that the model correctly classifies about 81% of test cases. The ROC AUC score of around 0.88 demonstrates strong discriminative ability.the model effectively separates the two classes across different probability thresholds. An AUC above 0.85 is generally considered good performance, suggesting the model has learned meaningful patterns from the training data.
Model Coefficients
The learned coefficients reveal how each feature influences the probability of passing.
Model Coefficients: Intercept: -1.685 Study Hours: 0.951 Previous Scores: 0.774
The positive coefficients for both study hours and previous scores confirm our intuition.more study time and higher previous scores both increase the probability of passing. Since we standardized the features, these coefficients are directly comparable. The study hours coefficient is larger in magnitude, suggesting it has a stronger influence on the outcome than previous scores in this dataset.
Sample Predictions
Let's examine predictions for a few individual students to see how the model assigns probabilities.
Sample Predictions: Hours: 3.3, Score: 95.9 -> Probability: 0.186 -> Predicted: Fail, Actual: Fail Hours: 5.3, Score: 58.9 -> Probability: 0.074 -> Predicted: Fail, Actual: Fail Hours: 2.2, Score: 89.0 -> Probability: 0.080 -> Predicted: Fail, Actual: Fail Hours: 1.0, Score: 87.8 -> Probability: 0.044 -> Predicted: Fail, Actual: Fail Hours: 7.2, Score: 76.8 -> Probability: 0.357 -> Predicted: Fail, Actual: Fail
These individual predictions show how the model translates feature values into probabilities. Notice that students with higher study hours and scores receive higher probabilities of passing. The model provides a probability for each student rather than just a binary prediction, which helps us assess its confidence in each prediction.
Detailed Evaluation Metrics
The classification report provides precision, recall, and F1-scores for each class, giving us deeper insight into model performance.
Classification Report:
precision recall f1-score support
0 0.83 0.94 0.88 159
1 0.52 0.27 0.35 41
accuracy 0.80 200
macro avg 0.68 0.60 0.62 200
weighted avg 0.77 0.80 0.77 200The classification report shows balanced performance across both classes. Precision indicates the proportion of predicted passes that were correct, while recall shows how many actual passes were correctly identified. The F1-score balances both metrics. The macro and weighted averages around 0.81 indicate consistent performance across both classes without significant bias toward either.
Confusion Matrix: [[149 10] [ 30 11]]
The confusion matrix shows the breakdown of correct and incorrect predictions. The diagonal elements represent correct predictions (true negatives and true positives), while off-diagonal elements show misclassifications. A well-performing model has most predictions concentrated on the diagonal.
Cross-Validation
Cross-validation provides a more robust estimate of model performance by evaluating on multiple train-test splits.
Cross-validation ROC AUC scores: [0.71725125 0.82343116 0.83082796 0.78262944 0.76960784] Mean CV ROC AUC: 0.785 (+/- 0.082)
The cross-validation scores show consistent performance across all five folds, with a mean AUC around 0.88. The small standard deviation indicates that the model's performance is stable and not dependent on a particular train-test split. This consistency suggests the model will generalize well to new data.
Polynomial Features Extension
For datasets with non-linear relationships, we can extend logistic regression by adding polynomial features to capture interactions and curved patterns.
from sklearn.preprocessing import PolynomialFeatures
# Create pipeline with polynomial features
poly_pipeline = make_pipeline(
StandardScaler(),
PolynomialFeatures(degree=2, include_bias=False),
LogisticRegression(random_state=42, max_iter=1000),
)
# Fit and evaluate
poly_pipeline.fit(X_train, y_train)
poly_accuracy = poly_pipeline.score(X_test, y_test)
poly_auc = roc_auc_score(y_test, poly_pipeline.predict_proba(X_test)[:, 1])Performance Comparison: Regular Logistic Regression - Accuracy: 0.800, ROC AUC: 0.762 Polynomial Logistic Regression - Accuracy: 0.805, ROC AUC: 0.760
In this example, adding polynomial features does not substantially improve performance because the underlying relationship is already approximately linear. The minimal improvement (less than 1%) likely reflects random variation rather than a better model. However, for datasets with true non-linear patterns or important feature interactions, polynomial features can provide significant performance gains. Use cross-validation to determine whether the added complexity is justified by improved generalization.
Key Parameters
Below are the main parameters that control how logistic regression works and performs.
penalty: Type of regularization to apply (default: 'l2'). Options include 'l1' (Lasso), 'l2' (Ridge), 'elasticnet', or 'none'. L1 performs feature selection, L2 handles multicollinearity, and elasticnet combines both.C: Inverse of regularization strength (default: 1.0). Smaller values specify stronger regularization. Values like 0.01 prevent overfitting with many features, while values like 10 or 100 allow closer fitting to training data. Typical range: [0.001, 0.01, 0.1, 1, 10, 100].solver: Algorithm for optimization (default: 'lbfgs'). Options include 'lbfgs' (good default), 'liblinear' (for small datasets), 'sag' and 'saga' (for large datasets), and 'newton-cg'. The 'saga' solver supports all penalty types.max_iter: Maximum number of iterations for convergence (default: 100). Increase to 1000 or more if you encounter convergence warnings. More iterations allow the optimizer more time to find the optimal solution.class_weight: Weights for classes (default: None). Set to 'balanced' to automatically adjust weights inversely proportional to class frequencies, which helps with imbalanced datasets.random_state: Seed for reproducibility (default: None). Set to an integer to ensure consistent results across runs, which is important for debugging and comparison.n_jobs: Number of CPU cores to use for parallel computation (default: None). Set to -1 to use all available cores, which speeds up cross-validation.
Key Methods
The following are the most commonly used methods for working with logistic regression models.
fit(X, y): Trains the logistic regression model on training data X and target labels y. This method learns the optimal coefficients through maximum likelihood estimation.predict(X): Returns predicted class labels (0 or 1) for input data X using the default 0.5 probability threshold. Use this when you need binary classifications.predict_proba(X): Returns probability estimates for each class. The output is an array where each row contains [probability of class 0, probability of class 1]. Use this when you need probability scores or want to apply custom thresholds.predict_log_proba(X): Returns log-probabilities for each class, which can be useful for numerical stability in certain applications or when working with log-likelihood calculations.score(X, y): Returns the mean accuracy on the given test data and labels. This is a convenience method that combines prediction and accuracy calculation.decision_function(X): Returns the signed distance of samples to the decision boundary (the linear predictor η before applying the logistic function). Useful for understanding how confident predictions are.