Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Linear Classifiers › Scikit-learn Implementation

Michael BrenndoerferApril 28, 20253 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 61Linear Classifiers › Scikit-learn Implementation

Reading section 10 of 16 · Scikit-learn Implementation · 3 minSection 10 of 16 · 3 min

Section 10 of 16 · 3 min read

Linear Classifiers › Scikit-learn ImplementationOne section · Sections in this chapter10 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

In practice, you will use scikit-learn's LogisticRegression for most tasks. It handles numerical stability, supports multiple solvers, and provides utilities for cross-validation and pipeline composition. Implementing gradient descent from scratch (as we did above) builds understanding, but scikit-learn's implementation uses more sophisticated optimization algorithms like L-BFGS that converge faster and more reliably.

In[21]:

Code

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

pipeline = Pipeline(
    [
        ("tfidf", TfidfVectorizer(max_features=50, stop_words="english")),
        ("clf", LogisticRegression(C=1.0, max_iter=500, random_state=42)),
    ]
)

X_docs = documents
y_docs = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

pipeline.fit(X_docs, y_docs)

Out[22]:

Console

Scikit-learn LogisticRegression - Training Results
==================================================
              precision    recall  f1-score   support

      sports       1.00      1.00      1.00         3
    politics       1.00      1.00      1.00         3
  technology       1.00      1.00      1.00         3

    accuracy                           1.00         9
   macro avg       1.00      1.00      1.00         9
weighted avg       1.00      1.00      1.00         9

Inspecting the Learned WeightsLink Copied

One advantage of linear classifiers over black-box models is interpretability. You can inspect which words (features) the model weighted most highly for each class.

In[23]:

Code

clf_sk = pipeline.named_steps["clf"]
tfidf_sk = pipeline.named_steps["tfidf"]
feature_names_sk = tfidf_sk.get_feature_names_out()
top_n = 5

Out[24]:

Console

Top weighted features per class:
=============================================

SPORTS
  victory        : +0.2421
  team           : +0.2421
  decisive       : +0.2421
  championship   : +0.2421
  winning        : +0.1977

POLITICS
  new            : +0.2580
  senator        : +0.2272
  officials      : +0.2272
  legislation    : +0.2272
  protection     : +0.2272

TECHNOLOGY
  learning       : +0.3217
  outperforms    : +0.1966
  previous       : +0.1966
  benchmarks     : +0.1966
  models         : +0.1966

The top-weighted words for each class match intuition. Sports-related words dominate the sports class, political vocabulary the politics class, and technical terms the technology class. When the model makes an error, inspecting the weights reveals which features drove the wrong decision. This level of transparency is a key advantage of linear classifiers in regulated domains where decisions must be explainable.

Visualizing the weight magnitudes across classes helps compare which features are most discriminative.

Out[25]:

Visualization

Heatmap showing logistic regression weight magnitudes for vocabulary features across three text classes. — Heatmap of logistic regression weights for the top 15 vocabulary features across three text categories (sports, politics, technology). Warm colors indicate features that strongly predict a class, while cool colors indicate features that work against a class. The clear row-wise structure confirms that each class has a distinct vocabulary signature, with minimal overlap.

Key ParametersLink Copied

The key parameters for sklearn.linear_model.LogisticRegression are:

C: Inverse regularization strength ( $C = 1/\lambda$ ). Smaller values apply stronger regularization. Defaults to 1.0.
penalty: Regularization type. 'l2' (default), 'l1', 'elasticnet', or 'none'.
solver: Optimization algorithm. 'lbfgs' works well for most cases; 'saga' supports L1 and is faster for large datasets.
max_iter: Maximum number of optimization iterations. Increase if the solver does not converge.
multi_class: 'ovr' (one-vs-rest) or 'multinomial' (softmax). 'multinomial' is usually better for multiclass tasks.