Linear Classifiers › Limitations of Linear Classifiers
Language AI HandbookAll chapters Linear Classifiers › Limitations of Linear ClassifiersOne section · Sections in this chapter14 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The defining constraint of linear classifiers is that they can only learn linear decision boundaries. In two dimensions, the boundary is a line. In three dimensions, a plane. In dimensions, a hyperplane. This is a fundamental limitation, not a matter of insufficient parameters. Adding more features to a linear classifier does not change the functional form of its decision boundary; it only adds dimensions to the space in which that flat boundary lives.
The classic example is XOR: four points labeled such that the pattern forms a checkerboard. No straight line can separate the positive from the negative examples correctly. The pattern requires a nonlinear boundary.
Logistic regression accuracy on XOR: 0.520 Baseline (predict majority class): 0.500 The linear classifier performs near chance on XOR. It cannot capture the diagonal separation pattern.
The linear classifier achieves accuracy near chance on this XOR problem. In NLP, analogous limitations appear in situations where meaning depends on word interactions. Detecting sarcasm, understanding negation ("not good"), or recognizing compositional phrases all require capturing nonlinear feature interactions that linear classifiers cannot represent.
Where Linear Boundaries Fail in Language
The limitations of linear classifiers are particularly visible in natural language understanding. Consider sentiment analysis with the word "not." In a bag-of-words representation, "not good" and "not bad" are treated as independent occurrences of "not," "good," and "bad." A linear classifier sees the same feature values and cannot distinguish between "The movie was not bad at all" (positive sentiment) and "The movie was not good" (negative sentiment), since both contain the word "not" and either "bad" or "good." Handling negation requires detecting the interaction between "not" and the sentiment word that follows it, a nonlinear operation.
Sarcasm presents a similar challenge. "Oh great, another meeting" is sarcastic and negative, but "Oh great, we won the game" is sincere and positive. The word "great" appears in both, but its meaning depends entirely on context. A linear classifier that weights "great" as a positive indicator will fail on the sarcastic case. It cannot consider the interaction between "great" and the surrounding context.
These are not edge cases but common patterns in real language. They explain why even high-quality logistic regression models on bag-of-words features typically plateau well below human performance on sentiment analysis benchmarks. The features needed to capture these patterns (like bigrams or trigrams capturing word sequences) can partially address negation, but the feature space explodes and the fundamental limitation remains.
The solution is introduced in the next chapter on Activation Functions and fully developed in the Multilayer Perceptrons chapter: stack linear classifiers with nonlinear activations between them. Each layer computes a linear transformation, then applies a nonlinear function, and the resulting composition can represent arbitrarily complex boundaries. Linear classifiers are not obsolete; they become the layers inside neural networks.
Feature Engineering as a Partial Remedy
Before deep learning, practitioners addressed the limitations of linear classifiers through feature engineering: manually designing features that make the relevant patterns more linearly separable. Adding bigrams (pairs of adjacent words) to the feature space lets the classifier learn that "not good" is a distinct feature from "good" appearing alone. Adding character n-grams helps with morphological variants. Adding handcrafted features like "does the sentence contain a negation word followed within three positions by a sentiment word?" can capture specific linguistic patterns directly.
This approach worked well enough to drive decades of NLP research, and feature-engineered linear models remained competitive well into the 2010s on many tasks. The key insight is that the transformation from raw text to features is itself encoding nonlinear structure, just in a hand-crafted rather than learned way. Neural networks automate this feature discovery, learning representations that make the final classification problem linearly easy. The final layer of a neural network classifier is always a linear layer with softmax. Deep learning did not abolish linear classifiers; it put them at the end of a learned feature hierarchy.