TF-IDF › The TF-IDF Formula
Language AI HandbookAll chapters TF-IDF › The TF-IDF FormulaOne section · Sections in this chapter2 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The TF-IDF formula answers two questions simultaneously for every word in every document: how important is this word within this specific document, and how distinctive is this word across the entire collection? A word becomes a strong signal only when it's both prominent locally and distinctive globally. The formula operationalizes this through multiplication of two complementary quantities.
Combining Term Frequency and Inverse Document Frequency
As covered in the previous two chapters, term frequency measures how often term appears in document , and inverse document frequency measures how rare is across the corpus .
The TF-IDF score combines these two signals through multiplication:
where:
- : the term (word) being evaluated
- : the document being examined
- : the full corpus (collection of all documents)
- : local prominence of in (e.g., raw count)
- : global rarity of across , where and is the number of documents containing
Why multiplication specifically? The four possible combinations reveal the logic. A word with high TF and low IDF (such as "the" appearing 50 times in a document) is locally dominant but globally ubiquitous. Multiplying a large TF by a near-zero IDF keeps the product low, correctly reflecting that this word tells you almost nothing distinctive. A word with low TF and high IDF (such as "backpropagation" appearing just once in a long general survey) is globally rare but barely present in this particular document. The product stays moderate, correctly signaling a weak indicator. A word with high TF and high IDF (such as "neural" appearing 8 times in a paper specifically about neural networks, when most other papers in the corpus don't mention it) satisfies both conditions simultaneously. Multiplication yields a high score, exactly as you'd want. Finally, a word with low TF and low IDF is neither prominent nor distinctive, and the product stays near zero.
This four-way analysis shows why any other combination rule would fail. Addition would give high scores to high-TF words regardless of IDF, reintroducing the stopword problem. Using only TF would be the bag-of-words problem you started with. Using only IDF would ignore how prominently a term appears in the document at hand. Multiplication is the right combinator because it requires both conditions to be satisfied before awarding a high score.
The Basic Formula in Practice
Let's compute TF-IDF from scratch on a small corpus to see the formula in action before dealing with production-level libraries.
import re
# A diverse corpus covering different ML subfields
corpus = [
"Machine learning algorithms learn patterns from data. Learning from data is powerful.",
"Deep learning uses neural networks. Neural networks learn hierarchical representations.",
"Natural language processing extracts meaning from text. Text processing is essential for NLP.",
"Computer vision analyzes images. Image recognition uses deep learning techniques.",
"Reinforcement learning agents learn through rewards. Learning optimal policies is challenging.",
]
def tokenize(text):
return re.findall(r"\b[a-z]+\b", text.lower())
tokenized_docs = [tokenize(doc) for doc in corpus]
N = len(corpus)from collections import Counter
import numpy as np
# Compute raw term frequency for each document
def compute_tf(tokens):
return Counter(tokens)
# Compute document frequency (how many docs contain each term)
def compute_df(tokenized_corpus):
df = Counter()
for doc in tokenized_corpus:
df.update(set(doc))
return df
# Compute IDF for each term
def compute_idf(df, num_docs):
return {term: np.log(num_docs / freq) for term, freq in df.items()}
# Compute TF-IDF vector for one document
def compute_tfidf(tf, idf):
return {term: count * idf.get(term, 0) for term, count in tf.items()}
tf_docs = [compute_tf(doc) for doc in tokenized_docs]
df = compute_df(tokenized_docs)
idf = compute_idf(df, N)
tfidf_docs = [compute_tfidf(tf, idf) for tf in tf_docs]Doc 1: Machine learning algorithms learn patterns from da... data TF-IDF = 3.219 from TF-IDF = 1.833 machine TF-IDF = 1.609 algorithms TF-IDF = 1.609 Doc 2: Deep learning uses neural networks. Neural network... neural TF-IDF = 3.219 networks TF-IDF = 3.219 hierarchical TF-IDF = 1.609 representations TF-IDF = 1.609 Doc 3: Natural language processing extracts meaning from ... processing TF-IDF = 3.219 text TF-IDF = 3.219 natural TF-IDF = 1.609 language TF-IDF = 1.609 Doc 4: Computer vision analyzes images. Image recognition... computer TF-IDF = 1.609 vision TF-IDF = 1.609 analyzes TF-IDF = 1.609 images TF-IDF = 1.609 Doc 5: Reinforcement learning agents learn through reward... reinforcement TF-IDF = 1.609 agents TF-IDF = 1.609 through TF-IDF = 1.609 rewards TF-IDF = 1.609
Each document's top TF-IDF terms capture exactly what makes it distinctive. Document 1 is dominated by "machine" and "algorithms," while Document 2 is characterized by "neural" and "hierarchical." Shared words like "learning" and "data" appear in multiple documents, which depresses their IDF values and thus their TF-IDF scores. Even though "learning" might occur many times in a single document, its ubiquity across the full corpus keeps its score moderate. This is TF-IDF working as intended: rewarding the specific over the generic.
Why the Logarithm in IDF?
The logarithm in the IDF formula deserves a moment's attention because it's doing real work. The raw ratio measures how many documents could contain the term for every document that does. For a corpus of 1,000,000 documents, a term appearing in only 1 document has a ratio of 1,000,000, while a term appearing in 10 documents has a ratio of 100,000. Without the logarithm, the first term would receive a weight 10 times larger than the second. That extreme difference doesn't reflect our intuitions about relevance: a term appearing in 1 out of a million documents versus 10 out of a million documents is not ten times more informative in any practical sense.
The logarithm compresses this scale. versus , a much more moderate difference. More generally, the logarithm converts multiplicative differences in document frequency into additive differences in IDF, which scale gracefully with corpus size. This is the same reason entropy is measured in logarithms: information content grows logarithmically with the inverse probability.
The base of the logarithm doesn't matter much in practice, because different bases produce IDF values that are proportional to one another (differing only by a constant factor). Natural logarithm (), , and all yield valid TF-IDF systems. The choice affects the absolute scale of scores but not which documents rank higher for a given query.