Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

TF-IDF › TF-IDF Vector Computation

Michael BrenndoerferUpdated March 21, 20264 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 13TF-IDF › TF-IDF Vector Computation

Reading section 4 of 13 · TF-IDF Vector Computation · 4 minSection 4 of 13 · 4 min

Section 4 of 13 · 4 min read

TF-IDF › TF-IDF Vector ComputationOne section · Sections in this chapter4 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Individual TF-IDF scores for one term in one document are useful for illustration, but practical NLP applications require vectors. You need one score per vocabulary term for each document, organized into a matrix where rows are documents and columns are terms. This document-term matrix is the primary output of TF-IDF computation. It enables similarity search and classification, and it also supports clustering.

Building the Document-Term MatrixLink Copied

For a corpus of $N$ documents with vocabulary of size $V$ , the TF-IDF matrix is an $N \times V$ matrix where entry $(i, j)$ is the TF-IDF score of the $j$ -th vocabulary term in the $i$ -th document. Most entries are zero: a typical document uses only a small fraction of the full vocabulary. This sparsity is not a problem to be solved but a property to be exploited.

In[9]:

Code

def build_vocabulary(tokenized_corpus):
    """Extract the sorted vocabulary from all documents."""
    vocab = sorted(set(term for doc in tokenized_corpus for term in doc))
    return {term: idx for idx, term in enumerate(vocab)}


def compute_tfidf_matrix(
    tokenized_corpus, use_log_tf=False, use_smoothed_idf=False
):
    """
    Build the full TF-IDF document-term matrix.
    Returns (matrix, vocabulary).
    """
    vocab = build_vocabulary(tokenized_corpus)
    V = len(vocab)
    N = len(tokenized_corpus)

    # Compute document frequencies
    df = compute_df(tokenized_corpus)

    # Build matrix
    matrix = np.zeros((N, V))
    for i, doc in enumerate(tokenized_corpus):
        tf = compute_tf(doc)
        max_count = max(tf.values()) if tf else 1
        for term, count in tf.items():
            if term in vocab:
                # TF variant
                if use_log_tf:
                    tf_val = log_tf(count)
                else:
                    tf_val = count
                # IDF variant
                if use_smoothed_idf:
                    idf_val = smoothed_idf(df[term], N)
                else:
                    idf_val = np.log(N / df[term])
                matrix[i, vocab[term]] = tf_val * idf_val
    return matrix, vocab


tfidf_matrix, vocab = compute_tfidf_matrix(tokenized_docs)
vocab_list = sorted(vocab, key=vocab.get)

Out[10]:

Console

TF-IDF Matrix shape: (5, 38)
  5 documents, 38 vocabulary terms

Non-zero entries: 48
Sparsity: 74.74% of entries are zero

Most entries in the TF-IDF matrix are zero because most terms appear in only a few documents. This sparsity is typical and grows dramatically as the corpus scales. A vocabulary of 100,000 terms with 1 million documents would yield a matrix with 100 billion potential entries, but perhaps only 5-10 billion would be non-zero. At this scale, dense NumPy arrays are completely impractical; sparse storage formats like Compressed Sparse Row (CSR) are essential. Scikit-learn's TfidfVectorizer uses CSR format internally, allowing it to handle corpora with millions of documents efficiently.

The sparsity also has a useful implication for computation: when computing cosine similarity between two documents, only terms that appear in both documents contribute to the dot product. If most terms are zero in either vector, the dot product computation can skip those terms entirely, making similarity search much faster than the dense formulation would suggest.

Visualizing the TF-IDF MatrixLink Copied

A heatmap of the TF-IDF scores reveals how the matrix captures document identity.

Out[11]:

Visualization

Heatmap of TF-IDF matrix with 5 document rows and vocabulary columns, showing bright cells for topic-specific terms. — TF-IDF score heatmap for a 5-document corpus. Each row represents a document and each column a vocabulary term. Brighter cells indicate higher TF-IDF scores, showing that each document has a distinctive signature of high-scoring terms. Terms appearing in many documents (like 'learning') show lower scores across all rows, while document-specific terms receive higher values concentrated in one row.

Each document has a distinctive visual profile in the heatmap. The documents on deep learning and neural networks share some high-scoring terms but diverge on specifics. Documents covering completely different topics have profiles that barely overlap at all. This distinctiveness is exactly what TF-IDF is designed to produce, and it's what makes the matrix useful for retrieval: if two documents have similar profiles (similar high-scoring terms), they're likely about similar topics.

Notice that the heatmap reveals another structural property: the matrix is not just sparse, it's block-like. Terms that are distinctive to a topic cluster together and activate strongly for the documents covering that topic. This block structure is the foundation of topic modeling methods like Latent Semantic Analysis (LSA), which we'll encounter in later chapters.