Inverse Document Frequency › Worked Example: IDF from Scratch on a New Corpus
Language AI HandbookAll chapters Inverse Document Frequency › Worked Example: IDF from Scratch on a New CorpusOne section · Sections in this chapter11 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's walk through a complete IDF computation on a fresh corpus to consolidate everything we've covered. We'll use a small corpus of movie reviews to show how IDF looks outside the machine learning domain.
# A small movie review corpus
movie_corpus = [
"A brilliant film with outstanding performances and compelling storytelling.",
"Dull plot drags on forever. The acting was mediocre at best.",
"Stunning visuals and a breathtaking score make this a must-watch masterpiece.",
"A predictable romantic comedy with charm and warmth that grows on you.",
"Terrifying horror that builds genuine dread through atmosphere and sound design.",
"The cinematography is breathtaking. The director crafts tension masterfully.",
"A romantic story with heartfelt performances and memorable dialogue.",
"Boring and predictable. The plot never surprises. Acting was flat.",
]
movie_tokenized = [tokenize(doc) for doc in movie_corpus]
movie_N = len(movie_tokenized)
# Step 1: Compute document frequencies
movie_df = compute_document_frequency(movie_tokenized)
# Step 2: Compute standard IDF
movie_idf = compute_idf(movie_df, movie_N)
# Step 3: Compute all variants
movie_idf_variants = compute_idf_variants(movie_df, movie_N)Movie Review Corpus IDF Analysis (8 documents) ====================================================================== Top 15 Most Informative Terms (highest IDF): ------------------------------------------------------- Term DF IDF Type ------------------------------------------------------- brilliant 1 2.0794 unique compelling 1 2.0794 unique film 1 2.0794 unique outstanding 1 2.0794 unique storytelling 1 2.0794 unique at 1 2.0794 unique best 1 2.0794 unique drags 1 2.0794 unique dull 1 2.0794 unique forever 1 2.0794 unique mediocre 1 2.0794 unique make 1 2.0794 unique masterpiece 1 2.0794 unique must 1 2.0794 unique score 1 2.0794 unique Terms with Zero IDF (appear in all 8 documents): None Medium IDF Terms (0.3 < IDF < 1.5, appear in 2-6 docs): --------------------------------------------- performances DF=2 IDF=1.3863 acting DF=2 IDF=1.3863 on DF=2 IDF=1.3863 plot DF=2 IDF=1.3863 was DF=2 IDF=1.3863 breathtaking DF=2 IDF=1.3863 predictable DF=2 IDF=1.3863 romantic DF=2 IDF=1.3863
The histogram reveals something true of virtually every natural language corpus: the vast majority of vocabulary terms appear in only a small fraction of documents, clustering at high IDF values. A small number of common words cluster at low IDF values. This bimodal shape is the statistical signature of natural language, where Zipf's law produces many rare terms and few common ones.
This distribution has a practical implication: after you apply IDF weighting to a TF matrix, most of the weight in your document vectors will be concentrated on relatively rare terms. Common words, which dominate raw word counts, are suppressed toward zero. The signal carried by specific, topic-relevant vocabulary is amplified. This is exactly the transformation that makes TF-IDF useful for retrieval.