Linear Classifiers › Summary

Linear classifiers are the foundation of modern neural networks. Every neuron in a deep network performs a linear transformation followed by a nonlinear activation, the same core operation as logistic regression. Understanding how linear classifiers work, what they can and cannot learn, and how gradient descent trains them gives you the conceptual tools to understand every more complex architecture that follows.

The trajectory of this chapter mirrors the broader arc of neural network design. We started with the basic weighted sum, added the sigmoid to produce probabilities, extended to multiple classes with softmax, added regularization to prevent overfitting, and then confronted the fundamental barrier of linear separability. Each step adds something necessary while retaining the core dot-product computation. When we move to multilayer networks in the chapters ahead, we are not replacing this foundation but building on it, adding the capacity to learn representations that make linearly inseparable problems tractable.

The key ideas from this chapter:

• A linear classifier computes $z = \mathbf{w} \cdot \mathbf{x} + b$ and classifies based on the sign of $z$
• The weight vector $\mathbf{w}$ is perpendicular to the decision boundary and points toward the positive region; the bias $b$ translates the boundary parallel to itself
• The dot product measures alignment between the input and the weight vector, serving as a template-matching operation that reappears in transformer attention mechanisms
• Logistic regression applies the sigmoid function $\sigma(z) = 1/(1 + e^{-z})$ to convert the score into a probability, with the cross-entropy loss arising naturally from maximum likelihood estimation
• Softmax generalizes logistic regression to $K$ classes by computing $P(y=k \mid \mathbf{x}) = e^{z_k} / \sum_j e^{z_j}$; numerical stability requires subtracting the maximum logit before exponentiating
• The cross-entropy loss is the natural loss for probability outputs because its gradients are large when the model is confidently wrong, unlike mean squared error
• Gradient descent updates weights by the prediction error times the input: $\mathbf{w} \leftarrow \mathbf{w} - \eta (\hat{p} - y)\mathbf{x}$
• The cross-entropy loss is convex in the parameters, guaranteeing a unique global optimum that gradient descent will find
• Regularization (L1 or L2) prevents overfitting when features outnumber training examples; L2 shrinks all weights uniformly while L1 promotes sparsity
• The perceptron (1958) was the first trainable linear classifier; its convergence proof and failure on XOR directly shaped the history of neural network research
• Linear classifiers cannot model nonlinear boundaries; adding nonlinear activation functions, covered in the Activation Functions chapter, resolves this limitation and forms the basis of multilayer networks