Bag of Words › Using scikit-learn's CountVectorizer

Michael BrenndoerferUpdated March 24, 20255 min read
Language AI HandbookAll chapters
Part 2 of 67Classical Text Representations
Chapter 6Bag of Words › Using scikit-learn's CountVectorizer
Reading section 7 of 14 · Using scikit-learn's CountVectorizer · 5 minSection 7 of 14 · 5 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 6 of 513

Section 7 of 14 · 5 min read

Bag of Words › Using scikit-learn's CountVectorizerOne section · Sections in this chapter7 / 14 ↓
Sections in this chapter14
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

While understanding the internals is valuable, in practice you'll use scikit-learn's CountVectorizer. It handles tokenization, vocabulary building, and sparse matrix creation in a single, optimized package. Under the hood, CountVectorizer implements all the same steps we've built from scratch: tokenizing text, building a vocabulary, counting occurrences, and storing the result as a CSR sparse matrix. The difference is that it does all of this efficiently, with built-in support for vocabulary pruning parameters and other preprocessing options.

One important practical detail is the fit/transform distinction. You fit the vectorizer on your training data to build the vocabulary. You then transform new documents using that same fixed vocabulary. This separation preserves the vector dimensions used during training: when you deploy a model in production, you cannot modify the vocabulary based on new documents you see at inference time. Any word in a new document that wasn't in the training vocabulary is simply ignored.

In[31]:
Code
from sklearn.feature_extraction.text import CountVectorizer

# Create vectorizer with common settings
vectorizer = CountVectorizer(
    lowercase=True,  # Convert to lowercase
    min_df=1,  # Minimum document frequency
    max_df=1.0,  # Maximum document frequency (fraction)
    binary=False,  # Use counts, not binary
    ngram_range=(1, 1),  # Unigrams only (single words)
)

# Fit and transform the corpus
bow_matrix = vectorizer.fit_transform(corpus)

# Get the vocabulary
feature_names = vectorizer.get_feature_names_out()
Out[32]:
Console
CountVectorizer Results:
  Matrix shape: (5, 22)
  Matrix type: <class 'scipy.sparse._csr.csr_matrix'>
  Vocabulary size: 22

Vocabulary (feature names):
  ['ai', 'and', 'are', 'artificial', 'deep', 'industries', 'intelligence', 'is', 'language', 'learning', 'machine', 'natural', 'networks', 'neural', 'of', 'power', 'processing', 'subset', 'systems', 'techniques', 'transforming', 'uses']

Sparse matrix info:
  Non-zero elements: 33
  Sparsity: 70.0%

CountVectorizer automatically handles tokenization, lowercasing, and vocabulary construction. The result is a sparse CSR matrix ready for machine learning. No manual preprocessing is required.

In[33]:
Code
# Convert to dense for visualization (only for small matrices!)
dense_matrix = bow_matrix.toarray()
Out[34]:
Visualization
Heatmap showing the document-term matrix with 5 documents and 22 vocabulary words, with cell colors indicating word counts.
Document-term matrix produced by scikit-learn's CountVectorizer for our five machine learning corpus documents. Each row is one document and each column is a vocabulary word. The vectorizer handled tokenization, lowercasing, and vocabulary construction automatically, producing a matrix ready for downstream classifiers.

Key Parameters

CountVectorizer offers extensive customization through its constructor parameters. Understanding what each parameter does allows you to tune the vocabulary construction process for your specific domain and dataset. The right parameter choices can dramatically improve downstream model performance by keeping informative words and discarding noise.

In[35]:
Code
# Demonstrate key parameters
vectorizers = {
    "default": CountVectorizer(),
    "binary": CountVectorizer(binary=True),
    "min_df=2": CountVectorizer(min_df=2),
    "max_df=0.8": CountVectorizer(max_df=0.8),
    "bigrams": CountVectorizer(ngram_range=(1, 2)),
}

results = {}
for name, vec in vectorizers.items():
    matrix = vec.fit_transform(corpus)
    results[name] = {
        "vocab_size": len(vec.get_feature_names_out()),
        "nnz": matrix.nnz,
        "sample_features": list(vec.get_feature_names_out()[:5]),
    }
Out[36]:
Console
CountVectorizer Parameter Comparison:
======================================================================

default:
  Vocabulary size: 22
  Non-zero elements: 33
  Sample features: ['ai', 'and', 'are', 'artificial', 'deep']

binary:
  Vocabulary size: 22
  Non-zero elements: 33
  Sample features: ['ai', 'and', 'are', 'artificial', 'deep']

min_df=2:
  Vocabulary size: 6
  Non-zero elements: 17
  Sample features: ['deep', 'is', 'learning', 'machine', 'of']

max_df=0.8:
  Vocabulary size: 21
  Non-zero elements: 28
  Sample features: ['ai', 'and', 'are', 'artificial', 'deep']

bigrams:
  Vocabulary size: 44
  Non-zero elements: 62
  Sample features: ['ai', 'ai and', 'and', 'and machine', 'are']

Each parameter setting produces different results. The binary setting keeps the same vocabulary but changes counts to presence indicators. Setting min_df=2 dramatically reduces vocabulary by removing rare words. The ngram_range=(1, 2) setting includes both unigrams and bigrams, capturing two-word phrases like "machine learning" and "deep learning". This dramatically increases vocabulary size but can capture meaningful phrases.

The key parameters for CountVectorizer are:

  • lowercase (default: True): Convert all text to lowercase before tokenizing. Set to False if case carries meaning (for example, proper nouns or acronyms).
  • min_df: Minimum document frequency threshold. If an integer, the word must appear in at least this many documents. If a float between 0.0 and 1.0, represents a proportion of documents. Use min_df=2 or higher to remove rare words and typos.
  • max_df: Maximum document frequency threshold. Words appearing in more than this fraction of documents are excluded. Use max_df=0.9 to remove extremely common words that provide no discriminative power.
  • binary (default: False): If True, all non-zero counts are set to 1. Use binary representation when word presence matters more than frequency.
  • ngram_range (default: (1, 1)): Tuple specifying the range of n-gram sizes to include. (1, 2) includes unigrams and bigrams, capturing phrases like "machine learning". Higher values dramatically increase vocabulary size.
  • stop_words: Either 'english' for built-in stop word list, or a custom list of words to exclude. Removes common words like "the", "is", "and" that typically add noise.
  • max_features: Limit vocabulary to the top N most frequent terms. Useful for controlling dimensionality in very large corpora.