Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Linear Classifiers › The Core Idea: Weighted Voting

Michael BrenndoerferApril 28, 20256 min read

Language AI HandbookAll chapters

Part 10 of 67Neural Network Foundations

Chapter 61Linear Classifiers › The Core Idea: Weighted Voting

Reading section 2 of 16 · The Core Idea: Weighted Voting · 6 minSection 2 of 16 · 6 min

Section 2 of 16 · 6 min read

Linear Classifiers › The Core Idea: Weighted VotingOne section · Sections in this chapter2 / 16 ↓

Sections in this chapter16

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

At its heart, a linear classifier is a voting system. Each input feature casts a vote, weighted by its importance, and the votes are summed to produce a decision score. Consider spam detection: words like "free" and "winner" should push the score toward spam, while words like "meeting" and "quarterly" should push it toward legitimate email. The classifier does not think about these words in the way a human reader does. It simply accumulates a numerical score, and the accumulated value determines the outcome.

This weighted voting metaphor reveals something important about the nature of linear models: they treat features independently. The contribution of "free" to the spam score does not depend on what other words are present in the email. The classifier cannot detect that "free" is less suspicious when it appears in "free of charge for your order confirmation" (a transactional email) versus "claim your FREE prize now" (spam). That kind of contextual reasoning requires feature interactions that linear classifiers cannot represent, a point we return to in the limitations section. But for many problems, independent feature voting works remarkably well.

Linear Classifier

A linear classifier predicts a class based on a linear combination of input features. For an input vector $\mathbf{x} \in \mathbb{R}^d$ , a weight vector $\mathbf{w} \in \mathbb{R}^d$ , and a bias scalar $b \in \mathbb{R}$ , the classifier computes the score $z = \mathbf{w} \cdot \mathbf{x} + b$ and predicts the positive class if $z > 0$ , otherwise the negative class.

The computation is a dot product plus a bias term. Given input $\mathbf{x} = [x_1, x_2, \ldots, x_d]^T$ and weights $\mathbf{w} = [w_1, w_2, \ldots, w_d]^T$ , the score is:

z = \mathbf{w} \cdot \mathbf{x} + b = \sum_{i=1}^{d} w_i x_i + b

where:

$d$ : the dimensionality of the input (number of features)
$x_i$ : the $i$ -th input feature value
$w_i$ : the weight assigned to feature $i$ , determining its contribution
$b$ : the bias term, which shifts the decision threshold away from zero
$z$ : the resulting score, which can be any real number

Let us make this concrete with a movie review example. We represent each review as a vector of word counts, and define weights that express our prior belief about which words indicate sentiment.

In[3]:

Code

import numpy as np

# Features: [count of "great", count of "terrible", count of "loved", count of "boring"]
# Positive sentiment words get positive weights; negative words get negative weights
weights = np.array([0.8, -0.9, 0.7, -0.6])
bias = -0.2

# Three example reviews represented as feature vectors
review_1 = np.array([3, 0, 2, 0])  # "great" 3x, "loved" 2x
review_2 = np.array([0, 2, 0, 3])  # "terrible" 2x, "boring" 3x
review_3 = np.array([1, 1, 1, 1])  # one of each word


def classify(x, w, b):
    """Return (score, label) for a feature vector x."""
    score = np.dot(w, x) + b
    label = "positive" if score > 0 else "negative"
    return score, label

Out[4]:

Console

Review 1 (enthusiastic): score = 3.60  ->  positive
Review 2 (disappointed): score = -3.80  ->  negative
Review 3 (mixed): score = -0.20  ->  negative

Review 1, dominated by positive words, scores high and is classified as positive. Review 2, dominated by negative words, scores below zero and is classified as negative. Review 3 is close to zero, sitting near the decision boundary, which reflects its mixed sentiment. The weights encode both which features matter and in which direction they push the decision.

Notice how the bias term $b = -0.2$ creates a slight skepticism in the classifier: a review needs some positive signal before it gets a positive label, rather than defaulting to positive on a neutral input. You can think of the bias as encoding the prior: how likely is positive sentiment before looking at any evidence?

The Dot Product as SimilarityLink Copied

The dot product $\mathbf{w} \cdot \mathbf{x}$ has a geometric interpretation that becomes central as we progress through this book. It measures how aligned the input vector $\mathbf{x}$ is with the weight vector $\mathbf{w}$ . More precisely, for vectors of unit length, $\mathbf{w} \cdot \mathbf{x} = \|\mathbf{w}\| \|\mathbf{x}\| \cos\theta$ , where $\theta$ is the angle between them. When the two vectors point in similar directions ( $\theta$ small), the dot product is large and positive. When they point in opposite directions ( $\theta$ near $180°$ ), the dot product is large and negative. When they are orthogonal ( $\theta = 90°$ ), the dot product is zero.

You can think of the weight vector as encoding the "ideal positive example" for the classifier. An input that closely resembles this ideal template scores high. An input that is the opposite scores low. This template-matching interpretation carries through to the attention mechanism in transformers: the query-key dot product in self-attention performs the same similarity computation, scaled up across thousands of dimensions and computed in parallel across all token pairs. Linear classifiers and attention heads are, at their computational core, doing the same basic thing.

The dot product is also the operation at the heart of linear regression, support vector machines, and word embedding comparisons like cosine similarity. Understanding it at the level of linear classifiers creates an intuition that transfers across all of these tools.

Why the Weighted Sum WorksLink Copied

It may seem surprising that such a simple formula, a weighted sum of features, works well for tasks as complex as language classification. Several reasons explain its effectiveness.

First, in many real-world problems, a small number of features carry most of the predictive signal, and these features act roughly independently. For spam detection, the presence of certain words is highly predictive regardless of context, because spammers reliably use the same vocabulary. A weighted sum over a bag-of-words representation can capture this pattern very effectively.

Second, with enough features, even linear classifiers can approximate nonlinear decision surfaces in the original input space. TF-IDF weighting, n-grams, and other feature engineering transformations implicitly create features that make problems more linearly separable. Before the deep learning era, practitioners spent enormous effort designing feature sets that would make linear models competitive with more complex alternatives. These efforts succeeded often enough that logistic regression remained a strong baseline well into the era of neural networks.

Third, linear classifiers have strong theoretical guarantees. They converge to a global optimum when the training objective is convex (as cross-entropy loss is for logistic regression). They generalize well when regularized, with formal bounds on the gap between training and test performance. And they are interpretable: you can read off the weights to understand what the model learned. These properties make them valuable as baselines and as deployed models in high-stakes settings where transparency matters.