Bag of Words › Key Functions and Parameters
Language AI HandbookAll chapters Bag of Words › Key Functions and ParametersOne section · Sections in this chapter12 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When working with Bag of Words representations, CountVectorizer from scikit-learn is the primary tool. Knowing its parameters and their effects allows you to adapt the vocabulary construction process to your specific domain and scale.
The most important consideration is the min_df and max_df balance. Setting min_df too low keeps noisy rare words. Setting min_df too high removes informative but moderately rare terms. In practice, start with min_df=2 on small corpora and min_df=5 on larger ones, then tune based on downstream performance. Similarly, max_df=0.95 is a reasonable default that removes very common terms while preserving domain-specific high-frequency words.
The ngram_range parameter deserves careful consideration. Bigrams can capture important phrases like "machine learning", "not good", and "highly recommend" that unigrams miss entirely. However, they also introduce a huge number of uninformative combinations. A good practice is to start with unigrams, measure performance, and then try ngram_range=(1, 2) to see if bigrams help. If vocabulary size becomes unmanageable, combine bigrams with a stricter min_df or a max_features cap.