Linear Classifiers › Weights and Bias: What They Encode

The weight vector $\mathbf{w}$ has a direct interpretation. Each element $w_i$ tells the classifier how much feature $i$ contributes to the positive class decision. A large positive $w_i$ means feature $i$ strongly indicates the positive class. A large negative $w_i$ means feature $i$ strongly indicates the negative class. A $w_i$ near zero means feature $i$ is irrelevant for this decision.

For text classification with a bag-of-words representation (which you encountered in the Bag of Words chapter), the weights are word-level indicators. After training, you can inspect the top-weighted words to understand what the classifier has learned. For sentiment analysis, words like "excellent," "outstanding," and "recommend" will have large positive weights, while "awful," "disappointing," and "refund" will have large negative weights. This interpretability is one of the most practically valuable properties of linear classifiers. When a model makes an error, you can trace the error back to which features were responsible and understand whether the model's reasoning is coherent or whether the training data contains a misleading pattern.

The bias $b$ sets the base rate for the classifier. If you set $b$ to a large positive value, the classifier will predict the positive class for almost every input, regardless of features. This is appropriate when the positive class is very common in the training data and the prior probability is high. Conversely, a large negative bias makes the classifier predict the negative class by default unless features strongly push it positive. In a well-trained model, the bias adjusts to reflect class imbalance in the training set, effectively encoding a prior probability before any evidence is considered.

The behavior of linear classifiers changes significantly depending on whether the feature space is sparse or dense. With bag-of-words representations in NLP, feature vectors are typically very sparse: a document might contain only a few hundred unique words out of a vocabulary of tens of thousands. Most entries in $\mathbf{x}$ are zero. The dot product $\mathbf{w} \cdot \mathbf{x}$ then only involves the non-zero entries, and the classifier effectively sums votes only from words present in the document.

Dense feature representations, such as averaged word embeddings or sentence vectors from a pretrained encoder, give every dimension a non-zero value. The classifier must consider the full vector, and every dimension can affect the decision. Logistic regression on top of dense embeddings is a common pattern in modern NLP, used as a lightweight classification head on top of a frozen pretrained model. The embeddings provide rich features; the linear classifier provides a trainable decision layer. This separation of feature extraction from classification is a recurring architectural pattern throughout this book.