Bag of Words › Vocabulary Construction
Language AI HandbookAll chapters Bag of Words › Vocabulary ConstructionOne section · Sections in this chapter3 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Building a vocabulary seems straightforward at first glance, but real-world text introduces significant complications. Raw text varies in capitalization and punctuation and may contain numbers or contractions. These variations make vocabulary construction a surprisingly subtle problem. How do you handle punctuation? Should "cat" and "Cat" be the same word? What about "run", "running", and "ran"? What about rare words that appear only once across millions of documents? What about extremely common words like "the" that appear everywhere but say nothing distinctive about a document's content?
These are not academic questions. The answers dramatically affect the size of your vocabulary, the density of your vectors, and ultimately the quality of the models you build on top. A naive vocabulary that includes every unique character sequence as a separate word quickly grows to tens of millions of entries, most of which appear only once and contribute only noise. A carefully pruned vocabulary that retains only the most informative terms produces tighter, more discriminative representations that generalize better to new documents.
The vocabulary construction process involves several sequential decisions. First, you tokenize the text, splitting it into individual word units. Then you normalize those tokens, typically by lowercasing and stripping punctuation. Finally, you filter the resulting vocabulary using frequency thresholds to remove both extremely rare and extremely common terms. Each of these steps involves tradeoffs that depend on your specific task and domain.
From Corpus to Vocabulary
A corpus is a collection of documents. The vocabulary is the set of unique terms extracted from this corpus. Let's work with a slightly more realistic example:
corpus = [
"Machine learning is a subset of artificial intelligence.",
"Deep learning is a subset of machine learning.",
"Natural language processing uses machine learning techniques.",
"AI and machine learning are transforming industries.",
"Neural networks power deep learning systems.",
]
def tokenize(text):
"""Simple tokenization: lowercase and split on whitespace/punctuation."""
import re
# Convert to lowercase and extract word tokens
tokens = re.findall(r"\b[a-z]+\b", text.lower())
return tokens
# Tokenize all documents
tokenized_corpus = [tokenize(doc) for doc in corpus]
# Build vocabulary
all_tokens = []
for doc in tokenized_corpus:
all_tokens.extend(doc)
vocabulary = sorted(set(all_tokens))
vocab_size = len(vocabulary)Corpus size: 5 documents Total tokens: 36 Vocabulary size: 23 unique words Vocabulary: a, ai, and, are, artificial deep, industries, intelligence, is, language learning, machine, natural, networks, neural of, power, processing, subset, systems techniques, transforming, uses
The vocabulary is significantly smaller than the total token count because many words repeat across documents. This compression, from raw tokens to unique vocabulary terms, is a key characteristic of text data. In practice, this ratio becomes even more pronounced as the corpus grows: adding more documents adds fewer and fewer new vocabulary words because the most common words appear in the very first documents.
Word Frequency Analysis
Before finalizing the vocabulary, examining word frequencies helps identify potential issues. The distribution of word frequencies in natural language follows Zipf's Law, which states that the frequency of a word is roughly inversely proportional to its rank in the frequency table. The most common word appears about twice as often as the second most common word, three times as often as the third, and so on. This mathematical regularity directly affects vocabulary construction.
Zipf's Law means that any natural language corpus contains a small number of very high-frequency words (function words like "the", "is", "a") and a very large number of low-frequency words (specific nouns, technical terms, proper names). The high-frequency words consume disproportionate space in your vectors but carry little discriminative information, since they appear in nearly every document regardless of topic. The low-frequency words are often highly informative when they do appear, but appear so rarely that they may be noise rather than signal.
from collections import Counter
# Count word frequencies across the corpus
word_counts = Counter(all_tokens)
# Sort by frequency
sorted_counts = word_counts.most_common()Word Frequencies (descending): ----------------------------------- learning 6 ██████ machine 4 ████ is 2 ██ a 2 ██ subset 2 ██ of 2 ██ deep 2 ██ artificial 1 █ intelligence 1 █ natural 1 █ language 1 █ processing 1 █ uses 1 █ techniques 1 █ ai 1 █ and 1 █ are 1 █ transforming 1 █ industries 1 █ neural 1 █ networks 1 █ power 1 █ systems 1 █
The word "learning" appears 5 times, "machine" appears 4 times, but many words appear only once. This pattern, a few high-frequency words and many rare words, follows Zipf's Law and is characteristic of natural language. In practice, this means that the decision of where to draw the frequency cutoff for your vocabulary has a large impact on both vocabulary size and model quality.