Linear Classifiers › Linear Classifiers

Before studying the complex architectures of modern neural networks, it pays to understand their fundamental building block: the linear classifier. Every neuron in a deep network applies a linear classifier followed by a nonlinearity. Understanding linear classifiers thoroughly means that when you later study multilayer perceptrons and backpropagation, the mechanics will feel natural rather than mysterious.

Linear classifiers make predictions by computing a weighted sum of input features and comparing the result against a threshold. They draw straight lines (or, in higher dimensions, hyperplanes) to separate data into categories. This simplicity limits what they can model, but it also makes them fast, interpretable, and mathematically tractable. Most of the machinery for training neural networks was developed in the context of linear classifiers first, including gradient descent, the concept of a loss function, and regularization. A solid grip on these ideas at the linear level pays dividends throughout the rest of this book.

The story of linear classifiers is in many ways the story of how machine learning learned to learn. Before statistical learning, classifiers were hand-coded: a programmer would read documents and write rules like "if the word 'free' appears more than twice, flag as spam." These rules broke as soon as the world changed. Learning-based classifiers replaced hand-crafted rules with a process that adjusts parameters automatically from labeled examples. The linear classifier is the simplest instantiation of this idea, and it is still the right tool for many practical problems today.

In this chapter, you will build linear classifiers from scratch. You will learn how weights and biases define decision boundaries, why the dot product is the central operation, and how logistic regression turns a raw score into a probability. We will extend to multiclass problems with the softmax function, apply these ideas to text classification, and train a complete model with gradient descent. We then examine regularization, the perceptron as a historical precursor, the connection to maximum likelihood estimation, and what linear classifiers fundamentally cannot do. These limitations motivate the nonlinear architectures in the chapters ahead.