Bag of Words › Count vs. Binary Representations
Language AI HandbookAll chapters Bag of Words › Count vs. Binary RepresentationsOne section · Sections in this chapter5 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
So far, we've counted word occurrences. But sometimes presence matters more than frequency. In a binary representation, each cell contains 1 if the word appears in the document and 0 otherwise, regardless of how many times it appears.
The choice between count and binary representation depends on what the frequency signal means for your task. Counts carry information about emphasis: a document that uses the word "excellent" ten times is making a stronger statement than one that uses it once. But counts can also mislead. A very long document will have higher counts for almost every word simply because it's longer, not because those words are more important. This length bias is one reason why TF-IDF weighting (covered in the next chapter) was developed as an improvement over raw counts.
Binary representations have a natural interpretation: they answer the question "is this topic present in the document?" rather than "how intensely is this topic discussed?" For some classification tasks, the distinction matters. Classifying whether a document is about sports or cooking depends mostly on presence of domain vocabulary, and binary features work just as well as counts. For other tasks, like distinguishing a rave review from a mild positive review, the frequency of strong positive words matters, and counts carry more signal.
# Create word-to-index mapping for our vocabulary
word_to_idx = {word: idx for idx, word in enumerate(sorted(vocabulary))}
# Count representation
def to_count_vector(tokens, word_to_idx):
vector = np.zeros(len(word_to_idx))
for token in tokens:
if token in word_to_idx:
vector[word_to_idx[token]] += 1
return vector
# Binary representation
def to_binary_vector(tokens, word_to_idx):
vector = np.zeros(len(word_to_idx))
for token in tokens:
if token in word_to_idx:
vector[word_to_idx[token]] = 1 # Set to 1, don't increment
return vector
# Compare representations for a sample document
sample_doc = tokenized_corpus[
1
] # "deep learning is a subset of machine learning"
count_vec = to_count_vector(sample_doc, word_to_idx)
binary_vec = to_binary_vector(sample_doc, word_to_idx)Sample document: 'Deep learning is a subset of machine learning.' Tokens: ['deep', 'learning', 'is', 'a', 'subset', 'of', 'machine', 'learning'] Comparison of representations: -------------------------------------------------- Word Count Binary -------------------------------------------------- a 1 1 deep 1 1 is 1 1 learning 2 1 ← differs machine 1 1 of 1 1 subset 1 1
Notice that "learning" appears twice in this document. The count representation records 2, while the binary representation records 1. Which is better? It depends on the task. For document classification, binary representations often work as well as counts. For tasks where word frequency carries meaning (like authorship attribution), counts are more informative. In practice, the best approach is to treat this as a hyperparameter and evaluate both on held-out data.