Term Frequency › Boolean Term Frequency
Language AI HandbookAll chapters Term Frequency › Boolean Term FrequencyOne section · Sections in this chapter5 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
At the extreme end of diminishing returns, you can ignore count entirely and just record whether a term is present or absent:
Boolean weighting discards all frequency information. Two documents where "economy" appears once and 50 times respectively receive identical representations for that term. This might seem like throwing away useful information, but in many practical situations it's exactly the right choice.
Consider a dataset of tweets. A tweet is typically 20 to 30 words long. In such a short document, most meaningful words appear exactly once. The difference between a count of 1 and a count of 2 for "election" in a 25-word tweet is dominated by noise: the author may have rephrased a sentence or mentioned the word in passing. Boolean representation treats these as identical, which is appropriate. In contrast, for a 1000-word news article, the difference between mentioning "election" twice and ten times is informative and worth preserving.
Boolean weighting also reduces the influence of verbose writing styles. Some authors repeat key terms frequently as a rhetorical device, while others state a concept once and move on. Boolean representation treats these authors as equally focused on a topic, which is often the right behavior for topic classification.
When to use boolean TF: Boolean weighting is surprisingly effective for tasks where the presence or absence of a concept matters more than how much it's discussed. Short documents like tweets or product titles often have counts of 1 or 2 for any non-stopword, so count differences are uninformative. Boolean features are also useful when working with binary classifiers or when you want to reduce the effect of verbose documents that repeat key terms excessively. In binary text classification benchmarks, boolean features sometimes outperform raw counts because models trained on boolean features are less susceptible to the length bias problem.
The Bag of Words chapter introduced binary vs count representations. Boolean TF is the same as binary BoW, with the only distinction being a matter of framing: BoW as a representation versus TF as a weighting scheme applied to counts.