Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Linear Classifiers › The Geometry: Decision Boundaries

Michael BrenndoerferApril 28, 20255 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 61Linear Classifiers › The Geometry: Decision Boundaries

Reading section 3 of 16 · The Geometry: Decision Boundaries · 5 minSection 3 of 16 · 5 min

Section 3 of 16 · 5 min read

Linear Classifiers › The Geometry: Decision BoundariesOne section · Sections in this chapter3 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding linear classifiers geometrically reveals both their power and their limitations. In two dimensions, the decision boundary is a line. In three dimensions, it is a plane. In $d$ dimensions, it is a $(d-1)$ -dimensional hyperplane. The critical insight is that this boundary is always flat.

The decision boundary is defined by the equation:

\mathbf{w} \cdot \mathbf{x} + b = 0

where:

$\mathbf{w}$ : the weight vector (the normal to the hyperplane)
$\mathbf{x}$ : a point on the boundary
$b$ : the bias, which shifts the boundary parallel to itself (without $b$ , the boundary must pass through the origin)

On one side of this hyperplane, $\mathbf{w} \cdot \mathbf{x} + b > 0$ and the classifier predicts the positive class. On the other side, $\mathbf{w} \cdot \mathbf{x} + b < 0$ and it predicts the negative class.

The weight vector $\mathbf{w}$ is always perpendicular to the decision boundary and points toward the positive region. To see why: for any two points $\mathbf{x}_1$ and $\mathbf{x}_2$ that both lie on the decision boundary (so both satisfy $\mathbf{w} \cdot \mathbf{x} + b = 0$ ), their difference satisfies:

\mathbf{w} \cdot (\mathbf{x}_1 - \mathbf{x}_2) = (\mathbf{w} \cdot \mathbf{x}_1 + b) - (\mathbf{w} \cdot \mathbf{x}_2 + b) = 0 - 0 = 0

where:

$\mathbf{x}_1 - \mathbf{x}_2$ : any vector lying in the decision boundary hyperplane
The equation shows this vector is orthogonal to $\mathbf{w}$ , confirming $\mathbf{w}$ is the normal direction

Out[5]:

Visualization

2D scatter plot showing linear decision boundary, weight vector, positive and negative class clusters. — A linear classifier in 2D feature space. The decision boundary (solid neutral line) separates positive examples (blue circles) from negative examples (red crosses). The weight vector w (green arrow) is perpendicular to the boundary and points toward the positive region. The color gradient shows the raw classifier score, ranging from high positive values (blue) to high negative values (red), illustrating that score magnitude encodes distance from the boundary.

The score $\mathbf{w} \cdot \mathbf{x} + b$ gives more than just a class label: it also provides a measure of confidence. Points far from the boundary have large absolute scores, indicating the classifier is confident. Points close to the boundary have scores near zero, indicating uncertainty. This magnitude becomes important when we convert scores to probabilities.

The distance from a point $\mathbf{x}$ to the decision boundary has a precise geometric meaning. It equals $\frac{|\mathbf{w} \cdot \mathbf{x} + b|}{\|\mathbf{w}\|}$ , the absolute score divided by the norm of the weight vector. A point with a high absolute score is geometrically far from the boundary, explaining why large scores correspond to high confidence. This distance interpretation connects linear classifiers to support vector machines (SVMs), which find the linear boundary that maximizes the minimum distance to any training point. SVMs are essentially linear classifiers with a specific training objective (maximum margin) rather than cross-entropy loss.

Changing the Boundary Through Weight ChangesLink Copied

Training a linear classifier means finding the weight vector $\mathbf{w}$ and bias $b$ that place the decision boundary in a good location, separating the training classes as cleanly as possible. As $\mathbf{w}$ changes direction, the hyperplane rotates. As $b$ changes, the hyperplane translates parallel to itself. Together, these $d + 1$ degrees of freedom (one per feature plus the bias) determine the full family of linear boundaries.

An important observation is that scaling $\mathbf{w}$ and $b$ by the same constant does not change the decision boundary at all. Multiplying $\mathbf{w}$ and $b$ by 2 produces the same hyperplane, but the raw scores are twice as large, and the sigmoid transformation of those scores produces more extreme probability estimates. This means the norm $\|\mathbf{w}\|$ controls model confidence while the direction $\mathbf{w}/\|\mathbf{w}\|$ controls the boundary location. Regularization, which we discuss later, constrains $\|\mathbf{w}\|$ to prevent the model from becoming overconfident.