Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Linear Classifiers › Regularization

Michael BrenndoerferApril 28, 20255 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 61Linear Classifiers › Regularization

Reading section 9 of 16 · Regularization · 5 minSection 9 of 16 · 5 min

Section 9 of 16 · 5 min read

Linear Classifiers › RegularizationOne section · Sections in this chapter9 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Logistic regression can overfit when the number of features is large relative to the number of training examples. With TF-IDF features, the vocabulary can easily contain tens of thousands of words, yet training data may be limited. In such settings, the model can memorize the training set by assigning extreme weights to rare words that appear in certain training documents but are not informative for the task.

The intuition for why this happens is straightforward. If a word appears in only one training document and that document belongs to class A, the model can drive its weight toward positive infinity, making the presence of that word nearly deterministic evidence for class A. This works perfectly on the training set but fails on new documents because the word's appearance was coincidental, not predictive.

Regularization adds a penalty term to the loss function that discourages large weights. The two most common forms are:

L2 regularization (ridge): adds the squared weight norm, penalizing all large weights proportionally:

\mathcal{L}_{\text{reg}} = \mathcal{L} + \frac{\lambda}{2} \|\mathbf{w}\|^2_2

L1 regularization (Lasso): adds the sum of absolute weights, encouraging sparsity:

\mathcal{L}_{\text{reg}} = \mathcal{L} + \lambda \|\mathbf{w}\|_1

where:

$\lambda > 0$ : the regularization strength hyperparameter; larger values apply stronger shrinkage
$\|\mathbf{w}\|^2_2 = \sum_{i=1}^{d} w_i^2$ : the squared L2 norm
$\|\mathbf{w}\|_1 = \sum_{i=1}^{d} |w_i|$ : the L1 norm, which can drive weights exactly to zero

The gradient of the L2 penalty with respect to $\mathbf{w}$ is $\lambda \mathbf{w}$ , adding a term proportional to the current weight magnitude. This gives a modified update rule:

\mathbf{w} \leftarrow \mathbf{w} - \eta \left(\frac{\partial \mathcal{L}}{\partial \mathbf{w}} + \lambda \mathbf{w}\right) = (1 - \eta \lambda) \mathbf{w} - \eta \frac{\partial \mathcal{L}}{\partial \mathbf{w}}

where:

$(1 - \eta \lambda)$ : the decay factor, slightly less than 1, shrinking weights by a fraction each step
This is why L2 regularization is also called "weight decay": weights are decayed toward zero at each update

L1 regularization can drive some weights exactly to zero, effectively performing automatic feature selection. In NLP, where the vocabulary is large and most words are uninformative for a given task, L1 can produce sparse weight vectors that are easier to inspect and interpret. A sparse model trained on a 50,000-word vocabulary might only use 500 non-zero weights, making the classifier's reasoning highly transparent.

The choice between L1 and L2 regularization involves a tradeoff between sparsity and stability. L2 shrinks all weights smoothly and produces stable, dense weight vectors. L1 sets many weights exactly to zero but can be unstable when many features are correlated, because any one of several correlated features can serve the same purpose and L1 arbitrarily zeros out some of them. Elastic net regularization combines both, which offers a smooth interpolation between the two extremes.

In[17]:

Code

def train_logistic_regression_l2(
    X, y, learning_rate=0.1, n_epochs=100, lambda_l2=0.01, random_state=42
):
    """Train logistic regression with L2 regularization."""
    n_samples, n_features = X.shape
    w = np.zeros(n_features)
    b = 0.0
    losses = []

    for epoch in range(n_epochs):
        z = X @ w + b
        p_hat = sigmoid(z)

        base_loss = binary_cross_entropy(y, p_hat)
        l2_penalty = (lambda_l2 / 2.0) * np.dot(w, w)
        losses.append(base_loss + l2_penalty)

        error = p_hat - y
        grad_w = (X.T @ error) / n_samples + lambda_l2 * w
        grad_b = error.mean()

        w -= learning_rate * grad_w
        b -= learning_rate * grad_b

    return w, b, losses

In[18]:

Code

lambdas = [0.0, 0.01, 0.1, 1.0]
results_reg = {}

for lam in lambdas:
    w_reg, b_reg, losses_reg = train_logistic_regression_l2(
        X_binary, y_binary, learning_rate=0.5, n_epochs=200, lambda_l2=lam
    )
    preds_reg = (sigmoid(X_binary @ w_reg + b_reg) >= 0.5).astype(int)
    acc_reg = (preds_reg == y_binary).mean()
    results_reg[lam] = {
        "w": w_reg,
        "b": b_reg,
        "losses": losses_reg,
        "accuracy": acc_reg,
        "weight_norm": np.linalg.norm(w_reg),
    }

Out[19]:

Console

  Lambda    Accuracy    Weight L2 Norm
--------------------------------------
   0.000       1.000            6.5942
   0.010       1.000            4.6584
   0.100       1.000            1.4709
   1.000       1.000            0.2012

As $\lambda$ increases, the weight norm decreases: regularization is working. On this tiny dataset, all settings achieve the same accuracy, but on a larger dataset with more features, differences in weight norm translate into differences in generalization.

Out[20]:

Visualization

Bar chart showing weight L2 norm decreasing as regularization strength increases. — Effect of L2 regularization strength on weight magnitudes. As the regularization parameter lambda increases from 0 to 1.0, the L2 norm of the weight vector shrinks substantially, with the most dramatic reduction between lambda=0 and lambda=0.1. Stronger regularization prevents large weights from forming on individual features, reducing overfitting risk at the cost of some model capacity.

Choosing the Regularization StrengthLink Copied

The regularization hyperparameter $\lambda$ (or equivalently, scikit-learn's $C = 1/\lambda$ ) is one of the most important decisions when training logistic regression. Too little regularization and the model overfits; too much and the model underfits, being unable to capture any real signal. The right value depends on the dataset size, the number of features, and the noise level.

Cross-validation is the standard approach to choosing $\lambda$ . You train models with a range of $\lambda$ values, evaluate each on a held-out validation set, and pick the value that produces the best validation performance. In scikit-learn, LogisticRegressionCV automates this process. In practice, it is worth checking whether regularization substantially changes your results: if it does, you are in a high-variance regime where more training data or stronger regularization would help. If it does not, you are likely in a high-bias regime where the model architecture itself is the limiting factor.