Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Inverse Document Frequency › The Information Theory Connection

Michael BrenndoerferMarch 28, 20257 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 12Inverse Document Frequency › The Information Theory Connection

Reading section 4 of 14 · The Information Theory Connection · 7 minSection 4 of 14 · 7 min

Section 4 of 14 · 7 min read

Inverse Document Frequency › The Information Theory ConnectionOne section · Sections in this chapter4 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

We've motivated IDF through intuition about word informativeness, but there's a deeper theoretical foundation. The IDF formula isn't arbitrary; it emerges naturally from information theory. This connection is one of the reasons IDF has held up so well over decades: it is a principled measurement.

Surprisal and Information ContentLink Copied

In information theory, the self-information (also called surprisal) of an event measures how surprising or informative that event is. The key insight is that rare events carry more information than common events. If someone tells you "the sun rose this morning," you learn nothing new. If they tell you "there was a solar eclipse this morning," you've learned something significant.

Self-Information

In information theory, the self-information (or surprisal) of an event with probability $p$ is:

I(p) = -\log(p) = \log\left(\frac{1}{p}\right)

where:

$p$ : the probability of the event occurring
$I(p)$ : the information content in bits (if using log base 2) or nats (if using natural log)

Rare events (low $p$ ) have high information content. Common events (high $p$ ) have low information content. An event with probability 1 has zero information content.

The intuition is elegant. If you already know something will happen with certainty, being told it happened conveys zero information. If something happens that you thought had only a 1-in-a-million chance, being told about it tells you an enormous amount. The logarithm formalizes this intuition: information grows as the reciprocal of probability, and the log makes the scale manageable.

From Probability to IDFLink Copied

Now let's connect this to document frequency. If we treat each document as a random draw from the corpus, we can estimate the probability that a randomly selected document contains term $t$ :

p(t) \approx \frac{\text{df}(t)}{N}

where:

$p(t)$ : the probability that a randomly selected document contains term $t$
$\text{df}(t)$ : the document frequency of term $t$ (how many documents contain it)
$N$ : the total number of documents in the corpus

This is simply the fraction of documents containing the term. A word appearing in 20 out of 100 documents has an estimated probability of $20/100 = 0.2$ of appearing in any given document.

Now we can substitute this probability estimate into the self-information formula and derive IDF step by step:

Step 1: Start with the self-information formula from above:

I(t) = \log\left(\frac{1}{p(t)}\right)

Step 2: Substitute our probability estimate $p(t) = \text{df}(t)/N$ :

I(t) = \log\left(\frac{1}{\text{df}(t)/N}\right)

Step 3: Simplify by flipping the fraction inside the logarithm (dividing by a fraction equals multiplying by its reciprocal):

I(t) = \log\left(\frac{N}{\text{df}(t)}\right)

Step 4: Recognize that this is exactly the IDF formula:

I(t) = \text{idf}(t)

This derivation shows that IDF is exactly the self-information of a term's occurrence. We're not using an arbitrary weighting scheme; we're measuring the information content of words in a principled way grounded in information theory.

This connection explains why IDF works so well. Information theory tells us that rare events are more informative, and IDF operationalizes this principle for text. Giving higher weights to rare words measures how much information they convey about document identity. A word appearing in 1% of documents has surprisal $\log(100) = 4.6$ nats. A word appearing in 50% of documents has surprisal $\log(2) = 0.69$ nats. The difference is not a designer's choice, it's the mathematically correct measure of how much information each word carries.

Let's verify this connection empirically by computing both IDF and self-information independently and comparing the results.

In[11]:

Code

# Step 1: Estimate probability of each term appearing in a document
def compute_probability(doc_freq, num_docs):
    """Estimate P(term appears in document) = df(t) / N."""
    return {term: df / num_docs for term, df in doc_freq.items()}


# Step 2: Compute self-information from probabilities
def compute_self_information(probabilities):
    """Compute self-information: I(t) = -log(P(t))."""
    return {term: -np.log(p) for term, p in probabilities.items()}


# Compute both quantities
probs = compute_probability(doc_freq, num_docs)
self_info = compute_self_information(probs)

Now we compare the self-information values (computed from probabilities) with the IDF values (computed from document frequencies). If our derivation is correct, they should be identical.

Out[12]:

Console

IDF as Self-Information:
======================================================================
Term                  P(term)      -log(P)          IDF     Match?
----------------------------------------------------------------------
  data                   0.40       0.9163       0.9163        yes
  deep                   0.40       0.9163       0.9163        yes
  from                   0.60       0.5108       0.5108        yes
  hierarchical           0.20       1.6094       1.6094        yes
  learn                  0.80       0.2231       0.2231        yes
  learning               0.80       0.2231       0.2231        yes
  networks               0.40       0.9163       0.9163        yes
  neural                 0.40       0.9163       0.9163        yes
  representations        0.20       1.6094       1.6094        yes
  uses                   0.60       0.5108       0.5108        yes

Every term shows a perfect match between self-information and IDF. Both calculations produce identical values. This isn't a coincidence or approximation; it's a mathematical identity. The IDF formula is the self-information formula, just expressed in terms of document counts rather than probabilities.

Out[13]:

Visualization

Curve showing self-information decreasing steeply as probability increases from 0 to 1. — Self-information as a function of probability. The curve shows that rare events (low p, near zero) carry extreme information content, while near-certain events (p near 1) carry almost none. The highlighted points at p=0.2, 0.5, 0.8, and 1.0 mark representative event probabilities.

Scatter plot with theoretical curve where corpus term IDF values follow the self-information formula. — IDF values for corpus terms plotted against their document frequency fraction. The scattered points fall exactly on the theoretical self-information curve I(p) = log(1/p), confirming that IDF computes the information content of a term's presence in a randomly selected document.

Why This Connection MattersLink Copied

The information-theoretic grounding is more than a theoretical curiosity. It tells you what IDF does and why. When you weight a word by its IDF, you're asking: "How surprising is it to encounter this word?" A word that appears in every document is never surprising, so it tells you nothing when you see it. A word that appears in one document in ten thousand is extremely surprising, meaning it carries a lot of signal about which document you're looking at.

This framing also suggests a natural interpretation for TF-IDF as a whole: it's asking "How much information does this word contribute to describing this particular document?" Term frequency captures how prominently the word features in the document (local signal), while IDF captures how surprising it is to see the word at all (global signal). Together, they form a complete picture.

The connection to information theory also helps when you're choosing between IDF variants or debugging retrieval systems. If IDF weights seem too extreme, you're asking: "Is the information content of seeing this word really that high?" If common words are dominating results, you're asking: "Why is a low-information word getting so much weight?" Having a theoretical framework makes these questions answerable.