Linear Classifiers › Feature Representation for Text
Language AI HandbookAll chapters Linear Classifiers › Feature Representation for TextOne section · Sections in this chapter7 / 16 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The linear classifier expects a fixed-length numerical vector. Text is not naturally numerical, so we need a representation strategy. As you learned in the Bag of Words and TF-IDF chapters, there are several options.
The simplest is binary bag-of-words: a vector of 0s and 1s indicating whether each vocabulary word appears in the document. A richer option is TF-IDF weighting, which downweights common words and upweights rare ones. The representation choice directly affects what the classifier can learn.
For a vocabulary of size , each document becomes a vector in . The classifier applies a weight to each vocabulary position. After training, the weight for word in class reflects how much seeing word increases the probability of class . This is interpretable: you can look at the highest-weighted words for each class and understand the classifier's reasoning.
The key insight is that the choice of text representation and the linear classifier are not independent decisions. They are two sides of the same coin. A TF-IDF representation is designed to make the signal for a linear classifier as strong as possible by reducing the noise from high-frequency stopwords. Feature engineering and model design are deeply intertwined for linear classifiers in a way that is less true for deep neural networks, which can learn their own representations.
from sklearn.feature_extraction.text import TfidfVectorizer
documents = [
# Sports (label 0)
"The team won the championship with a decisive victory in the final game",
"Baseball scores improved as the pitcher dominated the field all season long",
"Soccer players celebrated after winning the league title with stunning goals",
# Politics (label 1)
"The senator voted against the new healthcare legislation in congress",
"Presidential candidates debated economic policy and foreign affairs last week",
"Government officials announced new regulations for environmental protection",
# Technology (label 2)
"The new machine learning framework outperforms previous models on benchmarks",
"Software engineers deployed the neural network to the cloud computing platform",
"Deep learning algorithms process natural language with impressive accuracy",
]
labels = [0, 0, 0, 1, 1, 1, 2, 2, 2]
class_names_str = ["sports", "politics", "technology"]
vectorizer = TfidfVectorizer(max_features=30, stop_words="english")
X = vectorizer.fit_transform(documents).toarray()
vocab = vectorizer.get_feature_names_out()Vocabulary size: 30 Feature matrix shape: (9, 30) Sample vocabulary: ['learning', 'long', 'machine', 'models', 'natural', 'network', 'neural', 'new', 'officials', 'outperforms']
With 9 documents and a vocabulary of 30 words, each document becomes a 30-dimensional TF-IDF vector. The classifier will learn one weight per vocabulary word per class. In a real application, the vocabulary would be much larger, typically tens of thousands of words, and the weight matrix would have millions of entries. This is one reason why linear classifiers trained on bag-of-words features can be computationally intensive: not because of the classifier itself, but because of the high-dimensional sparse feature vectors they operate on.