Bag of Words › Vocabulary Pruning
Language AI HandbookAll chapters Bag of Words › Vocabulary PruningOne section · Sections in this chapter4 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Raw vocabularies from large corpora can contain millions of unique words. Many of these are noise: typos, rare technical terms, proper nouns that appear in only one document, or words so common that they appear everywhere. Vocabulary pruning removes uninformative terms to reduce dimensionality and improve model performance.
The case for pruning is both computational and statistical. On the computational side, every word in your vocabulary adds a dimension to every document vector, and operations over high-dimensional sparse vectors are expensive. On the statistical side, rare words that appear in only one or two documents provide almost no information for a classifier: the model cannot learn reliable patterns from a single or double observation. And common words that appear in nearly every document also provide no discriminative power, because a word that is equally likely in all classes tells you nothing about which class a document belongs to.
In practice, this means that the most informative words for text classification tend to be those in the middle of the frequency distribution: not so rare that they appear in only a handful of documents, and not so common that they appear everywhere. Vocabulary pruning attempts to focus the representation on exactly this informative middle zone.
Minimum Document Frequency
The document frequency of a word , denoted , counts how many distinct documents in the corpus contain that word. Notice that document frequency is different from term frequency: if a word appears 10 times in a single document, its term frequency is 10 but its document frequency is still just 1. Document frequency tells us how widespread a word is across the corpus, which is a better measure of a word's general informativeness than raw frequency.
Formally, document frequency is the size of the set of documents containing the word:
where:
- : the corpus (collection of all documents)
- : an individual document in the corpus
- : indicates that word appears in document
- : the count of elements in the set
Words that appear in very few documents provide little discriminative power and may represent noise. The min_df parameter sets a threshold: words must appear in at least this many documents (or this fraction of documents) to be included.
def compute_document_frequency(tokenized_corpus):
"""Count how many documents each word appears in."""
doc_freq = Counter()
for doc in tokenized_corpus:
# Count each word once per document
unique_words = set(doc)
doc_freq.update(unique_words)
return doc_freq
doc_freq = compute_document_frequency(tokenized_corpus)
# Apply min_df threshold
min_df = 2 # Word must appear in at least 2 documents
filtered_vocab_min = {word for word, freq in doc_freq.items() if freq >= min_df}Document Frequency Analysis: --------------------------------------------- Word Doc Freq Keep (min_df=2) --------------------------------------------- a 2 ✓ ai 1 ✗ and 1 ✗ are 1 ✗ artificial 1 ✗ deep 2 ✓ industries 1 ✗ intelligence 1 ✗ is 2 ✓ language 1 ✗ learning 5 ✓ machine 4 ✓ natural 1 ✗ networks 1 ✗ neural 1 ✗ of 2 ✓ power 1 ✗ processing 1 ✗ subset 2 ✓ systems 1 ✗ techniques 1 ✗ transforming 1 ✗ uses 1 ✗ --------------------------------------------- Original vocabulary: 23 words After min_df=2: 7 words
Words like "artificial", "industries", and "networks" appear in only one document. Removing them reduces our vocabulary while keeping words that appear across multiple documents. Notice that this pruning is conservative: we are only removing words that appear in a single document, which is the lowest possible threshold. In practice with large corpora, you might set min_df=5 or min_df=10 to be more aggressive.
Maximum Document Frequency
At the other extreme, words that appear in almost every document provide no discriminative power. The word "the" might appear in 95% of documents, making it useless for distinguishing between them. If "the" appears in sports articles, cooking articles, and science articles in roughly equal proportions, a classifier that learns to look for "the" gains nothing. The max_df parameter sets an upper threshold to filter out these ubiquitous words.
This is closely related to the concept of stop words, a predefined list of common words (like "the", "is", "and", "of") that are routinely excluded from text representations. The max_df approach is more principled than fixed stop word lists because it adapts to the corpus. In a specialized medical corpus, a word like "patient" might appear in 80% of documents and should be filtered out, even though it would not appear on any general-purpose stop word list.
# Apply max_df threshold (as a fraction of documents)
max_df = 0.8 # Word must appear in at most 80% of documents
num_docs = len(tokenized_corpus)
max_doc_count = int(max_df * num_docs)
filtered_vocab_max = {
word for word, freq in doc_freq.items() if freq <= max_doc_count
}
# Combined filtering
filtered_vocab = {
word
for word, freq in doc_freq.items()
if freq >= min_df and freq <= max_doc_count
}Total documents: 5 max_df = 0.8 → max document count = 4 Words appearing in >80% of documents: 'learning' appears in 5/5 documents Vocabulary after min_df=2, max_df=0.8: 6 words
In our small corpus, "learning" appears in all 5 documents (100%), exceeding our 80% threshold. In real applications, you might filter out words appearing in more than 90% of documents to remove uninformative terms like "the", "is", and "a". The combined effect of min_df and max_df pruning tends to keep precisely the words that carry the most discriminative information about document content.
Vocabulary Reduction with min_df
How aggressively should you prune? The answer depends on your corpus size and your task. For very large corpora with millions of documents, even a min_df of 5 or 10 removes a substantial fraction of the vocabulary while keeping all words with meaningful statistical support. For small corpora like our example, even min_df=2 has a significant effect.
Let's visualize how vocabulary size changes as we increase the min_df threshold:
# Calculate vocabulary size at different min_df thresholds
min_df_values = range(1, num_docs + 1)
vocab_sizes = []
for threshold in min_df_values:
remaining = sum(1 for freq in doc_freq.values() if freq >= threshold)
vocab_sizes.append(remaining)
The steep drop from min_df=1 to min_df=2 is typical. In real corpora, a large fraction of words appear only once (called hapax legomena, from the Greek for "said only once"). Removing these rare words often improves model performance by reducing noise without losing much signal. This is because hapax legomena are disproportionately likely to be misspellings, unusual proper names, or highly specialized jargon that appears in only one domain-specific document. Keeping them inflates the vocabulary and creates dimensions that are always zero except for that one document.