Linear Classifiers and Logistic Regression

Before studying the complex architectures of modern neural networks, it pays to understand their fundamental building block: the linear classifier. Every neuron in a deep network applies a linear classifier followed by a nonlinearity. Understanding linear classifiers thoroughly means that when you later study multilayer perceptrons and backpropagation, the mechanics will feel natural rather than mysterious.

Linear classifiers make predictions by computing a weighted sum of input features and comparing the result against a threshold. They draw straight lines (or, in higher dimensions, hyperplanes) to separate data into categories. This simplicity limits what they can model, but it also makes them fast, interpretable, and mathematically tractable. Most of the machinery for training neural networks was developed in the context of linear classifiers first, including gradient descent, the concept of a loss function, and regularization. A solid grip on these ideas at the linear level pays dividends throughout the rest of this book.

The story of linear classifiers is in many ways the story of how machine learning learned to learn. Before statistical learning, classifiers were hand-coded: a programmer would read documents and write rules like "if the word 'free' appears more than twice, flag as spam." These rules broke as soon as the world changed. Learning-based classifiers replaced hand-crafted rules with a process that adjusts parameters automatically from labeled examples. The linear classifier is the simplest instantiation of this idea, and it is still the right tool for many practical problems today.

In this chapter, you will build linear classifiers from scratch. You will learn how weights and biases define decision boundaries, why the dot product is the central operation, and how logistic regression turns a raw score into a probability. We will extend to multiclass problems with the softmax function, apply these ideas to text classification, and train a complete model with gradient descent. We then examine regularization, the perceptron as a historical precursor, the connection to maximum likelihood estimation, and what linear classifiers fundamentally cannot do. These limitations motivate the nonlinear architectures in the chapters ahead.

At its heart, a linear classifier is a voting system. Each input feature casts a vote, weighted by its importance, and the votes are summed to produce a decision score. Consider spam detection: words like "free" and "winner" should push the score toward spam, while words like "meeting" and "quarterly" should push it toward legitimate email. The classifier does not think about these words in the way a human reader does. It simply accumulates a numerical score, and the accumulated value determines the outcome.

This weighted voting metaphor reveals something important about the nature of linear models: they treat features independently. The contribution of "free" to the spam score does not depend on what other words are present in the email. The classifier cannot detect that "free" is less suspicious when it appears in "free of charge for your order confirmation" (a transactional email) versus "claim your FREE prize now" (spam). That kind of contextual reasoning requires feature interactions that linear classifiers cannot represent, a point we return to in the limitations section. But for many problems, independent feature voting works remarkably well.

The computation is a dot product plus a bias term. Given input $\mathbf{x} = [x_1, x_2, \ldots, x_d]^T$ and weights $\mathbf{w} = [w_1, w_2, \ldots, w_d]^T$, the score is:

where:

• $d$: the dimensionality of the input (number of features)
• $x_i$: the $i$-th input feature value
• $w_i$: the weight assigned to feature $i$, determining its contribution
• $b$: the bias term, which shifts the decision threshold away from zero
• $z$: the resulting score, which can be any real number

Let us make this concrete with a movie review example. We represent each review as a vector of word counts, and define weights that express our prior belief about which words indicate sentiment.

Review 1, dominated by positive words, scores high and is classified as positive. Review 2, dominated by negative words, scores below zero and is classified as negative. Review 3 is close to zero, sitting near the decision boundary, which reflects its mixed sentiment. The weights encode both which features matter and in which direction they push the decision.

Notice how the bias term $b = -0.2$ creates a slight skepticism in the classifier: a review needs some positive signal before it gets a positive label, rather than defaulting to positive on a neutral input. You can think of the bias as encoding the prior: how likely is positive sentiment before looking at any evidence?

The dot product $\mathbf{w} \cdot \mathbf{x}$ has a geometric interpretation that becomes central as we progress through this book. It measures how aligned the input vector $\mathbf{x}$ is with the weight vector $\mathbf{w}$. More precisely, for vectors of unit length, $\mathbf{w} \cdot \mathbf{x} = \|\mathbf{w}\| \|\mathbf{x}\| \cos\theta$, where $\theta$ is the angle between them. When the two vectors point in similar directions ($\theta$ small), the dot product is large and positive. When they point in opposite directions ($\theta$ near $180°$), the dot product is large and negative. When they are orthogonal ($\theta = 90°$), the dot product is zero.

You can think of the weight vector as encoding the "ideal positive example" for the classifier. An input that closely resembles this ideal template scores high. An input that is the opposite scores low. This template-matching interpretation carries through to the attention mechanism in transformers: the query-key dot product in self-attention performs the same similarity computation, scaled up across thousands of dimensions and computed in parallel across all token pairs. Linear classifiers and attention heads are, at their computational core, doing the same basic thing.

The dot product is also the operation at the heart of linear regression, support vector machines, and word embedding comparisons like cosine similarity. Understanding it at the level of linear classifiers creates an intuition that transfers across all of these tools.

It may seem surprising that such a simple formula, a weighted sum of features, works well for tasks as complex as language classification. Several reasons explain its effectiveness.

First, in many real-world problems, a small number of features carry most of the predictive signal, and these features act roughly independently. For spam detection, the presence of certain words is highly predictive regardless of context, because spammers reliably use the same vocabulary. A weighted sum over a bag-of-words representation can capture this pattern very effectively.

Second, with enough features, even linear classifiers can approximate nonlinear decision surfaces in the original input space. TF-IDF weighting, n-grams, and other feature engineering transformations implicitly create features that make problems more linearly separable. Before the deep learning era, practitioners spent enormous effort designing feature sets that would make linear models competitive with more complex alternatives. These efforts succeeded often enough that logistic regression remained a strong baseline well into the era of neural networks.

Third, linear classifiers have strong theoretical guarantees. They converge to a global optimum when the training objective is convex (as cross-entropy loss is for logistic regression). They generalize well when regularized, with formal bounds on the gap between training and test performance. And they are interpretable: you can read off the weights to understand what the model learned. These properties make them valuable as baselines and as deployed models in high-stakes settings where transparency matters.

Understanding linear classifiers geometrically reveals both their power and their limitations. In two dimensions, the decision boundary is a line. In three dimensions, it is a plane. In $d$ dimensions, it is a $(d-1)$-dimensional hyperplane. The critical insight is that this boundary is always flat.

The decision boundary is defined by the equation:

where:

• $\mathbf{w}$: the weight vector (the normal to the hyperplane)
• $\mathbf{x}$: a point on the boundary
• $b$: the bias, which shifts the boundary parallel to itself (without $b$, the boundary must pass through the origin)

On one side of this hyperplane, $\mathbf{w} \cdot \mathbf{x} + b > 0$ and the classifier predicts the positive class. On the other side, $\mathbf{w} \cdot \mathbf{x} + b < 0$ and it predicts the negative class.

The weight vector $\mathbf{w}$ is always perpendicular to the decision boundary and points toward the positive region. To see why: for any two points $\mathbf{x}_1$ and $\mathbf{x}_2$ that both lie on the decision boundary (so both satisfy $\mathbf{w} \cdot \mathbf{x} + b = 0$), their difference satisfies:

where:

• $\mathbf{x}_1 - \mathbf{x}_2$: any vector lying in the decision boundary hyperplane
• The equation shows this vector is orthogonal to $\mathbf{w}$, confirming $\mathbf{w}$ is the normal direction

The score $\mathbf{w} \cdot \mathbf{x} + b$ gives more than just a class label: it also provides a measure of confidence. Points far from the boundary have large absolute scores, indicating the classifier is confident. Points close to the boundary have scores near zero, indicating uncertainty. This magnitude becomes important when we convert scores to probabilities.

The distance from a point $\mathbf{x}$ to the decision boundary has a precise geometric meaning. It equals $\frac{|\mathbf{w} \cdot \mathbf{x} + b|}{\|\mathbf{w}\|}$, the absolute score divided by the norm of the weight vector. A point with a high absolute score is geometrically far from the boundary, explaining why large scores correspond to high confidence. This distance interpretation connects linear classifiers to support vector machines (SVMs), which find the linear boundary that maximizes the minimum distance to any training point. SVMs are essentially linear classifiers with a specific training objective (maximum margin) rather than cross-entropy loss.

Training a linear classifier means finding the weight vector $\mathbf{w}$ and bias $b$ that place the decision boundary in a good location, separating the training classes as cleanly as possible. As $\mathbf{w}$ changes direction, the hyperplane rotates. As $b$ changes, the hyperplane translates parallel to itself. Together, these $d + 1$ degrees of freedom (one per feature plus the bias) determine the full family of linear boundaries.

An important observation is that scaling $\mathbf{w}$ and $b$ by the same constant does not change the decision boundary at all. Multiplying $\mathbf{w}$ and $b$ by 2 produces the same hyperplane, but the raw scores are twice as large, and the sigmoid transformation of those scores produces more extreme probability estimates. This means the norm $\|\mathbf{w}\|$ controls model confidence while the direction $\mathbf{w}/\|\mathbf{w}\|$ controls the boundary location. Regularization, which we discuss later, constrains $\|\mathbf{w}\|$ to prevent the model from becoming overconfident.

The weight vector $\mathbf{w}$ has a direct interpretation. Each element $w_i$ tells the classifier how much feature $i$ contributes to the positive class decision. A large positive $w_i$ means feature $i$ strongly indicates the positive class. A large negative $w_i$ means feature $i$ strongly indicates the negative class. A $w_i$ near zero means feature $i$ is irrelevant for this decision.

For text classification with a bag-of-words representation (which you encountered in the Bag of Words chapter), the weights are word-level indicators. After training, you can inspect the top-weighted words to understand what the classifier has learned. For sentiment analysis, words like "excellent," "outstanding," and "recommend" will have large positive weights, while "awful," "disappointing," and "refund" will have large negative weights. This interpretability is one of the most practically valuable properties of linear classifiers. When a model makes an error, you can trace the error back to which features were responsible and understand whether the model's reasoning is coherent or whether the training data contains a misleading pattern.

The bias $b$ sets the base rate for the classifier. If you set $b$ to a large positive value, the classifier will predict the positive class for almost every input, regardless of features. This is appropriate when the positive class is very common in the training data and the prior probability is high. Conversely, a large negative bias makes the classifier predict the negative class by default unless features strongly push it positive. In a well-trained model, the bias adjusts to reflect class imbalance in the training set, effectively encoding a prior probability before any evidence is considered.

The behavior of linear classifiers changes significantly depending on whether the feature space is sparse or dense. With bag-of-words representations in NLP, feature vectors are typically very sparse: a document might contain only a few hundred unique words out of a vocabulary of tens of thousands. Most entries in $\mathbf{x}$ are zero. The dot product $\mathbf{w} \cdot \mathbf{x}$ then only involves the non-zero entries, and the classifier effectively sums votes only from words present in the document.

Dense feature representations, such as averaged word embeddings or sentence vectors from a pretrained encoder, give every dimension a non-zero value. The classifier must consider the full vector, and every dimension can affect the decision. Logistic regression on top of dense embeddings is a common pattern in modern NLP, used as a lightweight classification head on top of a frozen pretrained model. The embeddings provide rich features; the linear classifier provides a trainable decision layer. This separation of feature extraction from classification is a recurring architectural pattern throughout this book.

The raw score $z = \mathbf{w} \cdot \mathbf{x} + b$ can be any real number, from negative infinity to positive infinity. This makes it hard to interpret as a probability, since probabilities must lie between 0 and 1. Logistic regression solves this by passing the score through the sigmoid function.

To convert $z$ into a probability, we need a function that maps $\mathbb{R}$ to $(0, 1)$. The sigmoid function does exactly this:

where:

• $z$: the raw linear score $\mathbf{w} \cdot \mathbf{x} + b$, any real number
• $e^{-z}$: the exponential of $-z$; this quantity is large when $z$ is negative and small when $z$ is positive
• $1 + e^{-z}$: the denominator, which normalizes the output
• $\sigma(z) \in (0, 1)$: the resulting probability, always strictly between 0 and 1

The sigmoid function has three key properties:

• When $z \to +\infty$: $e^{-z} \to 0$, so $\sigma(z) \to 1$ (confident positive prediction)
• When $z \to -\infty$: $e^{-z} \to \infty$, so $\sigma(z) \to 0$ (confident negative prediction)
• When $z = 0$: $\sigma(0) = \frac{1}{1+1} = 0.5$ (maximum uncertainty, on the decision boundary)

The function is smooth and differentiable everywhere, a requirement for gradient descent. Its derivative also has a convenient form:

where:

• $\sigma(z)$: the sigmoid output at $z$
• $1 - \sigma(z)$: one minus the sigmoid, since $\sigma(z) + (1 - \sigma(z)) = 1$

This derivative is easy to compute once you have the forward pass value, which makes backpropagation through the sigmoid efficient. You compute $\sigma(z)$ on the forward pass and store it; on the backward pass, you compute $\sigma(z)(1 - \sigma(z))$ without any additional expensive operations.

Review 1 gets a high probability of being positive. Review 2 gets a low probability. Review 3, near the decision boundary, gets a probability close to 0.5. These probabilities are more informative than a hard yes/no decision because they communicate the model's confidence. A downstream system can decide how to act based on these probabilities: it might automatically file a very high-confidence prediction but route uncertain ones to a human reviewer.

Visualizing the sigmoid function helps build intuition about how it compresses scores into probabilities.

Notice how the sigmoid "saturates" in the tails: for $z > 4$ or $z < -4$, the function barely changes. This saturation has consequences for training. When the classifier is very confident but wrong, the gradient of the sigmoid is tiny, meaning learning happens slowly. This is the vanishing gradient problem in its simplest form, and it is one reason why alternative activations like ReLU (covered in the Activation Functions chapter) were developed for hidden layers of deep networks. For output layers producing probability estimates, however, sigmoid and softmax remain standard.

The sigmoid function has a natural probabilistic interpretation through log-odds. If $p = P(y=1 \mid \mathbf{x})$, then the odds in favor of the positive class are $p / (1-p)$, and the log-odds (also called the logit) are $\log(p / (1-p))$. Inverting the sigmoid relationship shows that:

This means logistic regression is modeling the log-odds of the positive class as a linear function of the input features. A unit increase in feature $i$ multiplies the odds of the positive class by $e^{w_i}$. This multiplicative relationship in odds space corresponds to an additive relationship in log-odds space, which is what makes the math tractable. The log-odds framing also connects logistic regression to Naive Bayes classifiers: under a specific set of distributional assumptions (that features are conditionally independent given the class), Naive Bayes produces a classifier with the same logistic regression form, with weights determined by feature likelihood ratios.

Binary classification (two classes) covers many problems, but NLP is full of multiclass problems: classifying documents into dozens of topics, assigning one of hundreds of part-of-speech tags, or selecting the next word from a vocabulary of 50,000 tokens. Logistic regression extends to multiple classes through the softmax function.

For $K$ classes, we define $K$ weight vectors $\mathbf{w}_1, \mathbf{w}_2, \ldots, \mathbf{w}_K$ and $K$ biases $b_1, b_2, \ldots, b_K$. The model computes one score per class. For class $k$ and input $\mathbf{x}$:

where:

• $\mathbf{w}_k \in \mathbb{R}^d$: the weight vector for class $k$, one per class
• $b_k$: the bias for class $k$
• $z_k$: the logit (raw score) for class $k$

The resulting score vector $\mathbf{z} = [z_1, z_2, \ldots, z_K]$ is called the vector of logits. To convert logits into a probability distribution, we apply the softmax function. For class $k$:

where:

• $e^{z_k}$: the exponential of the $k$-th class logit, which ensures all values are positive
• $\sum_{j=1}^{K} e^{z_j}$: the sum of exponentials across all $K$ classes, the normalizing constant
• $P(y = k \mid \mathbf{x}) \in (0, 1)$: the resulting probability for class $k$

The softmax output is a valid probability distribution: all values are positive (because exponentials are always positive) and they sum to exactly 1 (by construction). The class with the highest logit always receives the highest probability. The exponential function amplifies differences: if class $A$ has a logit 2 units above class $B$, then $A$ receives $e^2 \approx 7.4$ times the unnormalized probability mass of $B$.

The softmax function is ubiquitous in modern deep learning. Every language model that predicts the next token applies softmax over a vocabulary of tens of thousands of entries to produce a distribution over what comes next. Every classification network ends with a softmax layer. Understanding how softmax works at the scale of logistic regression gives you the intuition to understand what is happening at vastly larger scale in transformer models.

Computing $e^{z_k}$ directly causes overflow when logits are large (for example, when $z_k = 1000$, $e^{1000}$ exceeds the floating-point limit). The standard fix is to subtract the maximum logit before exponentiating. This does not change the result because, for any constant $c$:

where:

• $c = \max_j z_j$: the constant we subtract (chosen as the maximum so that the largest exponential is $e^0 = 1$)
• All other values $e^{z_j - c} \leq 1$: avoiding overflow

Setting $c = \max_j z_j$ ensures the numerically largest exponential equals 1 and all others are at most 1. This technique appears in every modern framework, including NumPy, PyTorch, and TensorFlow. If you ever implement softmax from scratch and forget this step, your code will silently produce NaN values on inputs with large logits.

The first document, mentioning "game," "server," and "team," gets split probability between sports and technology because it matches both class templates. The second document, mentioning "election" and "vote," is confidently classified as politics. This ambiguity in the first document reflects semantic ambiguity: "game server crashed during finals" could plausibly come from either a gaming article or a sports one. The model's uncertainty is well-calibrated.

When we have $K$ classes, the individual weight vectors $\mathbf{w}_1, \ldots, \mathbf{w}_K$ can be stacked as rows of a weight matrix $\mathbf{W} \in \mathbb{R}^{K \times d}$. The logit vector becomes:

This is a matrix-vector product: each row of $\mathbf{W}$ computes a dot product with $\mathbf{x}$, producing one score per class. This operation is the first linear layer of any neural network. In a deep network, you stack many such transformations, but each individual transformation has exactly this form. Recognizing this pattern across different scales of complexity is key to understanding neural network architecture.

The linear classifier expects a fixed-length numerical vector. Text is not naturally numerical, so we need a representation strategy. As you learned in the Bag of Words and TF-IDF chapters, there are several options.

The simplest is binary bag-of-words: a vector of 0s and 1s indicating whether each vocabulary word appears in the document. A richer option is TF-IDF weighting, which downweights common words and upweights rare ones. The representation choice directly affects what the classifier can learn.

For a vocabulary of size $V$, each document becomes a vector in $\mathbb{R}^V$. The classifier applies a weight to each vocabulary position. After training, the weight $w_{k,v}$ for word $v$ in class $k$ reflects how much seeing word $v$ increases the probability of class $k$. This is interpretable: you can look at the highest-weighted words for each class and understand the classifier's reasoning.

The key insight is that the choice of text representation and the linear classifier are not independent decisions. They are two sides of the same coin. A TF-IDF representation is designed to make the signal for a linear classifier as strong as possible by reducing the noise from high-frequency stopwords. Feature engineering and model design are deeply intertwined for linear classifiers in a way that is less true for deep neural networks, which can learn their own representations.

With 9 documents and a vocabulary of 30 words, each document becomes a 30-dimensional TF-IDF vector. The classifier will learn one weight per vocabulary word per class. In a real application, the vocabulary would be much larger, typically tens of thousands of words, and the weight matrix would have millions of entries. This is one reason why linear classifiers trained on bag-of-words features can be computationally intensive: not because of the classifier itself, but because of the high-dimensional sparse feature vectors they operate on.

So far, we have assumed the weights $\mathbf{w}$ and bias $b$ are given. In practice, we learn them from labeled training data by minimizing a loss function. The loss function quantifies how wrong the model's predictions are, and gradient descent iteratively adjusts the parameters to reduce that wrongness.

For binary classification, the standard loss function for logistic regression is the binary cross-entropy. Given a training example $(\mathbf{x}, y)$ where $y \in \{0, 1\}$ and the model predicts $\hat{p} = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$, the loss is:

where:

• $y \in \{0, 1\}$: the true label
• $\hat{p} = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$: the model's predicted probability for the positive class
• $-\log \hat{p}$: the active term when $y = 1$; this approaches 0 as $\hat{p} \to 1$ and approaches $+\infty$ as $\hat{p} \to 0$
• $-\log(1 - \hat{p})$: the active term when $y = 0$; this approaches 0 as $\hat{p} \to 0$ and approaches $+\infty$ as $\hat{p} \to 1$

The cross-entropy loss has an important property: it penalizes confident wrong predictions much more severely than uncertain ones. If the model assigns probability 0.01 to the true class, the loss is $-\log(0.01) \approx 4.6$. If it assigns 0.9, the loss is only $-\log(0.9) \approx 0.1$.

For multiclass problems with $K$ classes, the cross-entropy generalizes. The true label $y$ is represented as a one-hot vector $\mathbf{y} = [y_1, y_2, \ldots, y_K]$ where $y_k = 1$ if $k$ is the true class and $y_k = 0$ otherwise. The multiclass cross-entropy is:

where:

• $y_k \in \{0, 1\}$: the one-hot indicator for class $k$
• $P(y = k \mid \mathbf{x})$: the softmax probability for class $k$
• Since exactly one $y_k = 1$, this simplifies to $-\log P(y = \text{true class} \mid \mathbf{x})$: only the true class probability is penalized

You might wonder why we use cross-entropy loss instead of mean squared error (MSE). After all, MSE is simpler and widely used in regression. The issue is that MSE applied to probability outputs produces gradients that vanish when the model is very wrong. Consider a training example where the true label is $y = 1$ but the model predicts $\hat{p} = 0.001$ (extremely wrong). The MSE gradient is proportional to $(\hat{p} - y) \cdot \hat{p}(1-\hat{p})$, which is tiny because $\hat{p}(1-\hat{p})$ is tiny near 0. The model would learn almost nothing from this catastrophically wrong prediction. Cross-entropy avoids this: its gradient is $\hat{p} - y = 0.001 - 1 = -0.999$, which is large, producing a strong learning signal even when the model is very wrong.

This is one of the most important practical lessons in designing learning systems: the choice of loss function matters for what it measures and for the quality of the learning signal it provides. Cross-entropy is the natural loss for models that produce probability estimates, both because of its probabilistic derivation (negative log-likelihood) and because of its favorable gradient properties.

To minimize the loss, we need the gradient of $\mathcal{L}$ with respect to the weights $\mathbf{w}$ and bias $b$. A key result of differentiating through the sigmoid and logarithm is that the gradient has a remarkably simple form. For a single training example $(\mathbf{x}, y)$:

where:

• $\hat{p} - y$: the prediction error, the difference between the model's predicted probability and the true label
• $\mathbf{x}$: the input features, which scale the gradient for each individual weight dimension

The gradient formula has an elegant structure: it is exactly the prediction error times the input. If the model predicts $\hat{p} = 0.9$ but the true label is $y = 0$ (error = 0.9), the gradient is large and pushes the weights to reduce the score for this input. If the prediction is correct and confident ($\hat{p} = 0.95$, $y = 1$, error = 0.05), the gradient is small and the update is tiny. We only learn substantially from mistakes.

Deriving this gradient is a good exercise. Starting from $\mathcal{L} = -y \log \sigma(z) - (1-y)\log(1 - \sigma(z))$ with $z = \mathbf{w} \cdot \mathbf{x} + b$, and using the chain rule together with $\sigma'(z) = \sigma(z)(1-\sigma(z))$, the derivatives of the logarithms cancel beautifully with the derivative of the sigmoid, yielding exactly $\hat{p} - y$. This cancellation is not accidental: it is the reason that cross-entropy is the "right" loss for sigmoid outputs, a pairing that produces clean gradients and fast learning.

Gradient descent updates parameters iteratively:

where:

• $\eta > 0$: the learning rate, controlling step size
• $\frac{\partial \mathcal{L}}{\partial \mathbf{w}}$: the gradient pointing uphill; subtracting it moves downhill toward lower loss

For large datasets, computing gradients over the full training set is expensive. Stochastic gradient descent (SGD) uses a single random example or a small mini-batch per update, which is much faster and introduces useful noise that can help escape shallow local minima. The Stochastic Gradient Descent chapter covers SGD in depth.

One of the most important theoretical properties of logistic regression is that the cross-entropy loss is a convex function of the parameters $\mathbf{w}$ and $b$. Convexity means that the loss surface has no local minima: any local minimum is guaranteed to be the global minimum. Gradient descent on a convex function will always find the optimal solution, provided the learning rate is small enough and training runs long enough.

This guarantee does not hold for neural networks with hidden layers, which have non-convex loss surfaces with many local minima. Practitioners working with deep networks must worry about initialization, learning rate schedules, and optimization methods suited to these surfaces. With logistic regression, the optimization is straightforward. This makes logistic regression a reliable baseline and a sanity check: if your neural network performs worse than logistic regression on the same features, something has gone wrong with the training procedure.

The convexity also means that the solution is unique (assuming the data is not linearly separable with a finite margin). There is a single global optimum that all gradient descent trajectories converge to, regardless of initialization. This uniqueness is valuable for reproducibility and for understanding what the model has learned.

Logistic regression can overfit when the number of features is large relative to the number of training examples. With TF-IDF features, the vocabulary can easily contain tens of thousands of words, yet training data may be limited. In such settings, the model can memorize the training set by assigning extreme weights to rare words that appear in certain training documents but are not informative for the task.

The intuition for why this happens is straightforward. If a word appears in only one training document and that document belongs to class A, the model can drive its weight toward positive infinity, making the presence of that word nearly deterministic evidence for class A. This works perfectly on the training set but fails on new documents because the word's appearance was coincidental, not predictive.

Regularization adds a penalty term to the loss function that discourages large weights. The two most common forms are:

L2 regularization (ridge): adds the squared weight norm, penalizing all large weights proportionally:

L1 regularization (Lasso): adds the sum of absolute weights, encouraging sparsity:

where:

• $\lambda > 0$: the regularization strength hyperparameter; larger values apply stronger shrinkage
• $\|\mathbf{w}\|^2_2 = \sum_{i=1}^{d} w_i^2$: the squared L2 norm
• $\|\mathbf{w}\|_1 = \sum_{i=1}^{d} |w_i|$: the L1 norm, which can drive weights exactly to zero

The gradient of the L2 penalty with respect to $\mathbf{w}$ is $\lambda \mathbf{w}$, adding a term proportional to the current weight magnitude. This gives a modified update rule:

where:

• $(1 - \eta \lambda)$: the decay factor, slightly less than 1, shrinking weights by a fraction each step
• This is why L2 regularization is also called "weight decay": weights are decayed toward zero at each update

L1 regularization can drive some weights exactly to zero, effectively performing automatic feature selection. In NLP, where the vocabulary is large and most words are uninformative for a given task, L1 can produce sparse weight vectors that are easier to inspect and interpret. A sparse model trained on a 50,000-word vocabulary might only use 500 non-zero weights, making the classifier's reasoning highly transparent.

The choice between L1 and L2 regularization involves a tradeoff between sparsity and stability. L2 shrinks all weights smoothly and produces stable, dense weight vectors. L1 sets many weights exactly to zero but can be unstable when many features are correlated, because any one of several correlated features can serve the same purpose and L1 arbitrarily zeros out some of them. Elastic net regularization combines both, which offers a smooth interpolation between the two extremes.

The regularization hyperparameter $\lambda$ (or equivalently, scikit-learn's $C = 1/\lambda$) is one of the most important decisions when training logistic regression. Too little regularization and the model overfits; too much and the model underfits, being unable to capture any real signal. The right value depends on the dataset size, the number of features, and the noise level.

Cross-validation is the standard approach to choosing $\lambda$. You train models with a range of $\lambda$ values, evaluate each on a held-out validation set, and pick the value that produces the best validation performance. In scikit-learn, LogisticRegressionCV automates this process. In practice, it is worth checking whether regularization substantially changes your results: if it does, you are in a high-variance regime where more training data or stronger regularization would help. If it does not, you are likely in a high-bias regime where the model architecture itself is the limiting factor.

In practice, you will use scikit-learn's LogisticRegression for most tasks. It handles numerical stability, supports multiple solvers, and provides utilities for cross-validation and pipeline composition. Implementing gradient descent from scratch (as we did above) builds understanding, but scikit-learn's implementation uses more sophisticated optimization algorithms like L-BFGS that converge faster and more reliably.

One advantage of linear classifiers over black-box models is interpretability. You can inspect which words (features) the model weighted most highly for each class.

The key parameters for sklearn.linear_model.LogisticRegression are:

• C: Inverse regularization strength ($C = 1/\lambda$). Smaller values apply stronger regularization. Defaults to 1.0.
• penalty: Regularization type. 'l2' (default), 'l1', 'elasticnet', or 'none'.
• solver: Optimization algorithm. 'lbfgs' works well for most cases; 'saga' supports L1 and is faster for large datasets.
• max_iter: Maximum number of optimization iterations. Increase if the solver does not converge.
• multi_class: 'ovr' (one-vs-rest) or 'multinomial' (softmax). 'multinomial' is usually better for multiclass tasks.

Before logistic regression, the perceptron was the first trainable linear classifier, proposed by Frank Rosenblatt in 1958. The perceptron algorithm is simpler than gradient descent: it only updates weights when it makes a mistake.

The perceptron uses labels $y \in \{-1, +1\}$ (rather than $\{0, 1\}$) and predicts $\hat{y} = \text{sign}(\mathbf{w} \cdot \mathbf{x} + b)$. When the prediction is wrong (i.e., $\hat{y} \neq y$), it applies the update:

where:

• $y \in \{-1, +1\}$: the true class label
• $\mathbf{x}$: the input feature vector
• Adding $y \cdot \mathbf{x}$ to $\mathbf{w}$: when $y = +1$, moves $\mathbf{w}$ closer to $\mathbf{x}$, increasing the score; when $y = -1$, moves $\mathbf{w}$ away from $\mathbf{x}$, decreasing the score

The perceptron convergence theorem states that if the training data is linearly separable, the perceptron finds a separating hyperplane in a finite number of steps. The bound on the number of mistakes is $R^2 / \gamma^2$, where $R$ is the radius of the smallest sphere containing all training points and $\gamma$ is the margin (the distance from the closest training point to the optimal hyperplane). A larger margin means faster convergence. When there is no separating hyperplane, however, the perceptron loops indefinitely.

Logistic regression with gradient descent is strictly more powerful: it produces probability estimates rather than hard labels, converges to the best linear fit even on non-separable data, and minimizes a smooth convex loss function. The perceptron is primarily of historical and pedagogical interest today, but understanding it helps connect the update rules of modern neural network training to their historical roots. The perceptron's mistake-driven update rule also reappears in online learning settings where data arrives sequentially and one cannot afford to store the full training set.

The perceptron's announcement in 1958 generated enormous excitement. The New York Times ran a story about a machine that could learn. There was widespread belief that general artificial intelligence was imminent. Then, in 1969, Marvin Minsky and Seymour Papert published a thorough mathematical analysis showing that the perceptron could not solve XOR and other non-linearly-separable problems. The result was devastating to the field's optimism and contributed to the first "AI winter," a period of reduced funding and interest in neural network research. The solution, stacking perceptrons in multiple layers with nonlinear activations, was known even at the time, but efficient training algorithms for multilayer networks were not developed until the 1986 popularization of backpropagation. The lesson: architectural expressiveness without a training algorithm is not enough.

The cross-entropy loss is not an arbitrary choice. It arises naturally from maximum likelihood estimation (MLE), a foundational principle in statistics. Understanding this connection grounds the training process in probability theory and explains why cross-entropy is the "correct" objective.

Suppose we model the conditional probability $P(y = 1 \mid \mathbf{x}) = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$. Given a training dataset $\{(\mathbf{x}_1, y_1), \ldots, (\mathbf{x}_N, y_N)\}$ of independent examples, the likelihood of observing this dataset under our model is:

For binary labels, this becomes $\prod_{i=1}^{N} \hat{p}_i^{y_i} (1 - \hat{p}_i)^{1 - y_i}$. Taking the log (which turns the product into a sum and is monotonically increasing, so maximizing log-likelihood is equivalent to maximizing likelihood):

Minimizing the negative log-likelihood is exactly minimizing the average cross-entropy loss. Maximum likelihood estimation tells us that the cross-entropy loss is the principled objective for logistic regression: we are finding the parameters that make the observed training labels as likely as possible under our model. This connection is fundamental: it means that training logistic regression produces well-calibrated probability estimates, not just good rankings.

Well-calibrated probabilities are practically important. If your sentiment classifier says 90% probability of positive for a particular review, and it is well-calibrated, then roughly 90% of similar reviews with that predicted probability will indeed be positive. This means you can use the raw probability outputs as decision inputs, alongside the hard class predictions. Poor calibration, often seen in models trained with mean squared error or in overconfident deep networks, can lead to systematic errors in downstream systems that rely on probability values.

Before softmax became standard, and still useful in some settings, the one-vs-rest (OvR) strategy handles multiclass problems by training $K$ separate binary classifiers. Classifier $k$ is trained to distinguish class $k$ from all other classes. At prediction time, all $K$ classifiers produce a score, and the class with the highest score is predicted.

OvR is simple to implement: you just train $K$ logistic regression models independently, one for each class. Each classifier sees the full training set, with the positive class being the $k$-th class and the negative class being all other examples combined. This works well when the classes are well-separated and the models' output scores are on comparable scales.

The disadvantage of OvR is that the $K$ classifiers are not jointly trained to produce a coherent probability distribution. The scores from different classifiers are not comparable, and they do not necessarily sum to 1. Softmax (multinomial logistic regression) avoids this by training all $K$ weight vectors jointly and normalizing the outputs. In practice, multinomial logistic regression with softmax outperforms OvR, especially when classes are similar and the probability estimates need to be well-calibrated. Scikit-learn's multi_class='multinomial' parameter enables this.

The defining constraint of linear classifiers is that they can only learn linear decision boundaries. In two dimensions, the boundary is a line. In three dimensions, a plane. In $d$ dimensions, a hyperplane. This is a fundamental limitation, not a matter of insufficient parameters. Adding more features to a linear classifier does not change the functional form of its decision boundary; it only adds dimensions to the space in which that flat boundary lives.

The classic example is XOR: four points labeled such that the pattern forms a checkerboard. No straight line can separate the positive from the negative examples correctly. The pattern requires a nonlinear boundary.

The linear classifier achieves accuracy near chance on this XOR problem. In NLP, analogous limitations appear in situations where meaning depends on word interactions. Detecting sarcasm, understanding negation ("not good"), or recognizing compositional phrases all require capturing nonlinear feature interactions that linear classifiers cannot represent.

The limitations of linear classifiers are particularly visible in natural language understanding. Consider sentiment analysis with the word "not." In a bag-of-words representation, "not good" and "not bad" are treated as independent occurrences of "not," "good," and "bad." A linear classifier sees the same feature values and cannot distinguish between "The movie was not bad at all" (positive sentiment) and "The movie was not good" (negative sentiment), since both contain the word "not" and either "bad" or "good." Handling negation requires detecting the interaction between "not" and the sentiment word that follows it, a nonlinear operation.

Sarcasm presents a similar challenge. "Oh great, another meeting" is sarcastic and negative, but "Oh great, we won the game" is sincere and positive. The word "great" appears in both, but its meaning depends entirely on context. A linear classifier that weights "great" as a positive indicator will fail on the sarcastic case. It cannot consider the interaction between "great" and the surrounding context.

These are not edge cases but common patterns in real language. They explain why even high-quality logistic regression models on bag-of-words features typically plateau well below human performance on sentiment analysis benchmarks. The features needed to capture these patterns (like bigrams or trigrams capturing word sequences) can partially address negation, but the feature space explodes and the fundamental limitation remains.

The solution is introduced in the next chapter on Activation Functions and fully developed in the Multilayer Perceptrons chapter: stack linear classifiers with nonlinear activations between them. Each layer computes a linear transformation, then applies a nonlinear function, and the resulting composition can represent arbitrarily complex boundaries. Linear classifiers are not obsolete; they become the layers inside neural networks.

Before deep learning, practitioners addressed the limitations of linear classifiers through feature engineering: manually designing features that make the relevant patterns more linearly separable. Adding bigrams (pairs of adjacent words) to the feature space lets the classifier learn that "not good" is a distinct feature from "good" appearing alone. Adding character n-grams helps with morphological variants. Adding handcrafted features like "does the sentence contain a negation word followed within three positions by a sentiment word?" can capture specific linguistic patterns directly.

This approach worked well enough to drive decades of NLP research, and feature-engineered linear models remained competitive well into the 2010s on many tasks. The key insight is that the transformation from raw text to features is itself encoding nonlinear structure, just in a hand-crafted rather than learned way. Neural networks automate this feature discovery, learning representations that make the final classification problem linearly easy. The final layer of a neural network classifier is always a linear layer with softmax. Deep learning did not abolish linear classifiers; it put them at the end of a learned feature hierarchy.

Linear classifiers are the foundation of modern neural networks. Every neuron in a deep network performs a linear transformation followed by a nonlinear activation, the same core operation as logistic regression. Understanding how linear classifiers work, what they can and cannot learn, and how gradient descent trains them gives you the conceptual tools to understand every more complex architecture that follows.

The trajectory of this chapter mirrors the broader arc of neural network design. We started with the basic weighted sum, added the sigmoid to produce probabilities, extended to multiple classes with softmax, added regularization to prevent overfitting, and then confronted the fundamental barrier of linear separability. Each step adds something necessary while retaining the core dot-product computation. When we move to multilayer networks in the chapters ahead, we are not replacing this foundation but building on it, adding the capacity to learn representations that make linearly inseparable problems tractable.

The key ideas from this chapter:

• A linear classifier computes $z = \mathbf{w} \cdot \mathbf{x} + b$ and classifies based on the sign of $z$
• The weight vector $\mathbf{w}$ is perpendicular to the decision boundary and points toward the positive region; the bias $b$ translates the boundary parallel to itself
• The dot product measures alignment between the input and the weight vector, serving as a template-matching operation that reappears in transformer attention mechanisms
• Logistic regression applies the sigmoid function $\sigma(z) = 1/(1 + e^{-z})$ to convert the score into a probability, with the cross-entropy loss arising naturally from maximum likelihood estimation
• Softmax generalizes logistic regression to $K$ classes by computing $P(y=k \mid \mathbf{x}) = e^{z_k} / \sum_j e^{z_j}$; numerical stability requires subtracting the maximum logit before exponentiating
• The cross-entropy loss is the natural loss for probability outputs because its gradients are large when the model is confidently wrong, unlike mean squared error
• Gradient descent updates weights by the prediction error times the input: $\mathbf{w} \leftarrow \mathbf{w} - \eta (\hat{p} - y)\mathbf{x}$
• The cross-entropy loss is convex in the parameters, guaranteeing a unique global optimum that gradient descent will find
• Regularization (L1 or L2) prevents overfitting when features outnumber training examples; L2 shrinks all weights uniformly while L1 promotes sparsity
• The perceptron (1958) was the first trainable linear classifier; its convergence proof and failure on XOR directly shaped the history of neural network research
• Linear classifiers cannot model nonlinear boundaries; adding nonlinear activation functions, covered in the Activation Functions chapter, resolves this limitation and forms the basis of multilayer networks

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about linear classifiers.