Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multinomial Logistic Regression › Implementation in Scikit-learn

Michael BrenndoerferJune 15, 20258 min read

Machine Learning from ScratchAll chapters

Part 4 of 9Regression Models

Chapter 38Multinomial Logistic Regression › Implementation in Scikit-learn

Reading section 5 of 8 · Implementation in Scikit-learn · 8 minSection 5 of 8 · 8 min

Section 5 of 8 · 8 min read

Multinomial Logistic Regression › Implementation in Scikit-learnOne section · Sections in this chapter5 / 8 ↓

Sections in this chapter8

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Scikit-learn gives an efficient implementation of multinomial logistic regression through the LogisticRegression class. In this section, we'll walk through a complete example using a customer risk classification dataset.

Data PreparationLink Copied

First, we'll create our dataset and prepare it for modeling. We'll use age and income as predictors to classify customers into three risk categories:

In[7]:

Code

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder

# Create sample dataset
data = {
    "age": [25, 30, 35, 40, 45, 50, 25, 30, 35, 40, 45, 50],
    "income": [30, 40, 50, 60, 70, 80, 35, 45, 55, 65, 75, 85],
    "risk": [
        "low",
        "low",
        "low",
        "medium",
        "medium",
        "medium",
        "medium",
        "medium",
        "high",
        "high",
        "high",
        "high",
    ],
}

df = pd.DataFrame(data)

# Prepare features and target
X = df[["age", "income"]].values
y = df["risk"].values

# Encode categorical target variable
le = LabelEncoder()
y_encoded = le.fit_transform(y)

# Split into training and test sets (stratified to ensure all classes in both sets)
X_train, X_test, y_train, y_test = train_test_split(
    X, y_encoded, test_size=0.3, random_state=42, stratify=y_encoded
)

The LabelEncoder converts the categorical risk levels into numerical values (0, 1, 2), which is required for scikit-learn's implementation. We use stratified splitting to ensure all three risk categories appear in both training and test sets, which is important for reliable evaluation with small datasets.

Model TrainingLink Copied

Now let's train the multinomial logistic regression model using the L-BFGS solver, which is well-suited for small to medium-sized datasets:

In[8]:

Code

from sklearn.metrics import accuracy_score, log_loss

# Initialize and train the model
model = LogisticRegression(solver="lbfgs", max_iter=1000, random_state=42)
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)
y_pred_proba = model.predict_proba(X_test)

# Calculate performance metrics
test_accuracy = accuracy_score(y_test, y_pred)
train_accuracy = model.score(X_train, y_train)
logloss = log_loss(y_test, y_pred_proba)

Model CoefficientsLink Copied

Let's examine the learned coefficients for each class to understand how age and income influence risk classification:

Out[9]:

Console

Class Encoding:
  0: high
  1: low
  2: medium

Model Intercepts:
  Class 0 (high): 3.8853
  Class 1 (low): -1.6769
  Class 2 (medium): -2.2084

Model Coefficients:
  Class 0 (high):
    Age: -0.9481
    Income: 0.5568
  Class 1 (low):
    Age: 0.7673
    Income: -0.4956
  Class 2 (medium):
    Age: 0.1808
    Income: -0.0612

The coefficients reveal how each feature affects the log-odds of each class relative to the reference class (class 0, which corresponds to "high" risk in this encoding). Positive coefficients increase the probability of that class as the feature value increases, while negative coefficients decrease it. For instance, if the age coefficient for the "low" risk class is positive, older customers are more likely to be classified as low risk compared to the reference category. The intercepts represent the baseline log-odds when all features equal zero.

Model PerformanceLink Copied

Let's evaluate the model's performance on the test set:

Out[10]:

Console

Model Performance:
  Training Accuracy: 1.000
  Test Accuracy: 0.500
  Log Loss: 0.591

The test accuracy indicates how often the model correctly predicts the risk category. In this case, the model achieves reasonable accuracy despite the very small dataset size. The log loss measures the quality of the probability predictions.lower values indicate better calibrated probabilities, with values closer to 0 being preferable. The small difference between training and test accuracy (if any) suggests the model is not overfitting significantly, which is encouraging given the limited data.

Predictions and ProbabilitiesLink Copied

Let's examine individual predictions to understand how the model assigns probabilities:

Out[11]:

Console

Sample Predictions on Test Set:
--------------------------------------------------------------------------------
Observation 1:
  Features: Age=45, Income=75
  True Class: high
  Predicted Class: high
  Probabilities:
    high: 0.838
    low: 0.001
    medium: 0.161

Observation 2:
  Features: Age=35, Income=50
  True Class: low
  Predicted Class: medium
  Probabilities:
    high: 0.051
    low: 0.324
    medium: 0.625

Observation 3:
  Features: Age=40, Income=60
  True Class: medium
  Predicted Class: medium
  Probabilities:
    high: 0.109
    low: 0.100
    medium: 0.791

Observation 4:
  Features: Age=30, Income=45
  True Class: medium
  Predicted Class: high
  Probabilities:
    high: 0.456
    low: 0.106
    medium: 0.438

Each prediction shows the complete probability distribution across all three risk categories. The model assigns the observation to the class with the highest probability, but the full distribution provides valuable information about prediction confidence. For example, if one class has a probability of 0.9, the model is very confident, whereas probabilities of 0.4, 0.35, and 0.25 indicate high uncertainty. Notice how the probabilities sum to 1.0 by construction, which is a fundamental property of the softmax function used in multinomial logistic regression.

Classification ReportLink Copied

The classification report provides detailed performance metrics for each class:

Out[12]:

Console

Classification Report:
              precision    recall  f1-score   support

        high       0.50      1.00      0.67         1
         low       0.00      0.00      0.00         1
      medium       0.50      0.50      0.50         2

    accuracy                           0.50         4
   macro avg       0.33      0.50      0.39         4
weighted avg       0.38      0.50      0.42         4


Confusion Matrix:
[[1 0 0]
 [0 0 1]
 [1 0 1]]

Rows represent true classes, columns represent predicted classes

The classification report breaks down model performance by class:

Precision measures how many predicted instances of a class were correct. High precision means the model rarely misclassifies other categories as this one (few false positives).
Recall measures how many actual instances of a class were correctly identified. High recall means the model rarely misses instances of this class (few false negatives).
F1-score is the harmonic mean of precision and recall. It's useful when you want to balance the trade-off between precision and recall.
Support shows the number of actual instances of each class in the test set, which helps contextualize the other metrics.

The confusion matrix visualizes where the model makes mistakes. Each row represents the true class, and each column represents the predicted class. Diagonal elements (top-left to bottom-right) show correct predictions, while off-diagonal elements reveal specific misclassification patterns. For example, if the model frequently confuses "medium" risk with "high" risk, you'll see a large value in the corresponding off-diagonal position.

Note

The relatively low performance in this example is due to the very small dataset (only 12 observations). In practice, multinomial logistic regression requires substantially more data to learn meaningful patterns and achieve good generalization performance.

Key ParametersLink Copied

Below are the main parameters that control how multinomial logistic regression works in scikit-learn.

solver: Algorithm for optimization (default: 'lbfgs'). Choose 'lbfgs' or 'newton-cg' for small to medium datasets with L2 regularization. Use 'saga' for large datasets or when L1 regularization is needed. As of scikit-learn 1.5, the multinomial case is handled automatically. Avoid 'liblinear' for multinomial problems as it uses one-vs-rest instead.
penalty: Type of regularization (default: 'l2'). Options are 'l1', 'l2', 'elasticnet', or None. L2 regularization prevents overfitting by penalizing large coefficients. L1 can perform feature selection by driving some coefficients to zero. Note that not all solvers support all penalty types.
C: Inverse of regularization strength (default: 1.0). Smaller values specify stronger regularization. Start with 1.0 and adjust based on cross-validation. Common values to try: 0.01, 0.1, 1.0, 10.0, 100.0.
max_iter: Maximum number of iterations for solver convergence (default: 100). Increase if you get convergence warnings. Values of 1000 or higher may be needed for complex datasets or stochastic solvers.
random_state: Seed for reproducibility (default: None). Set to an integer (e.g., 42) to ensure consistent results across runs, especially important when using stochastic solvers like 'sag' or 'saga'.
class_weight: Weights for each class (default: None). Use 'balanced' to automatically adjust weights inversely proportional to class frequencies, which helps with imbalanced datasets. Can also provide a dictionary with custom weights.
tol: Tolerance for stopping criteria (default: 1e-4). The optimization stops when the change in coefficients or loss is smaller than this value. Smaller values lead to more precise solutions but longer training times.

Key MethodsLink Copied

The following methods are commonly used when working with multinomial logistic regression models.

fit(X, y): Trains the model on feature matrix X and target vector y. The target should be encoded as integers (use LabelEncoder for categorical targets).
predict(X): Returns predicted class labels for input data X. The model assigns each observation to the class with the highest predicted probability.
predict_proba(X): Returns probability estimates for each class. Output is an array where each row sums to 1.0, representing the probability distribution across all classes. Useful for understanding prediction confidence and setting custom decision thresholds.
predict_log_proba(X): Returns log-probabilities for each class. More numerically stable than predict_proba when working with very small probabilities.
score(X, y): Returns the mean accuracy on the given test data and labels. Provides a quick evaluation, though detailed metrics (precision, recall, F1) are often needed for multi-class problems.
decision_function(X): Returns the decision function values (linear predictors) for each class before applying the softmax transformation. Useful for understanding raw model outputs.