Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Term Frequency › Efficient Term Frequency Computation

Michael BrenndoerferMarch 30, 20254 min read

Language AI HandbookAll chapters

Part 2 of 67Classical Text Representations

Chapter 11Term Frequency › Efficient Term Frequency Computation

Reading section 11 of 17 · Efficient Term Frequency Computation · 4 minSection 11 of 17 · 4 min

Section 11 of 17 · 4 min read

Term Frequency › Efficient Term Frequency ComputationOne section · Sections in this chapter11 / 17 ↓

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When working with large corpora, computing and storing term frequency matrices efficiently requires more than just calling .fit_transform() and hoping for the best.

Sparse Matrix FormatsLink Copied

scikit-learn returns term frequency matrices as Compressed Sparse Row (CSR) matrices, which store only non-zero values and their positions. A CSR matrix consists of three arrays:

.data: the non-zero values
.indices: the column index of each non-zero value
.indptr: pointers indicating where each row's non-zeros start in .data

Understanding the CSR format helps you reason about when sparse operations are fast and when they may degrade to dense performance. Operations that iterate over all entries of a row are efficient in CSR format. Operations that require column-wise access (like accessing all documents that contain a specific term) are slower because the column indices are not contiguous in memory, which is why CSC (Compressed Sparse Column) format is sometimes preferred for column-oriented access patterns.

In[14]:

Code

from sklearn.feature_extraction.text import CountVectorizer

# Build a count matrix
vec = CountVectorizer(min_df=2)
X_sparse = vec.fit_transform(texts)
X_dense = X_sparse.toarray()

Out[15]:

Console

Matrix shape: (500, 7656)
Non-zero entries: 74,055

Memory usage:
  Dense matrix:   29906.2 KB
  Sparse matrix:  869.8 KB
  Compression:    34.4x smaller

CSR components:
  .data:    74,055 non-zero values
  .indices: 74,055 column indices
  .indptr:  501 row pointers

Computing L2 Normalization EfficientlyLink Copied

For large matrices, computing L2 norms row-by-row in Python is slow. scikit-learn provides sklearn.preprocessing.normalize which calls optimized C routines and operates directly on sparse matrices without expanding them to dense form:

In[16]:

Code

import time

from sklearn.preprocessing import normalize

# L2 normalization on a large sparse matrix
X_log = X_sparse.astype(float).copy()
# Apply log scaling to non-zero values (modifying only .data)
X_log.data = 1 + np.log(X_log.data)

# sklearn's normalize works directly on sparse matrices
start = time.perf_counter()
X_log_l2 = normalize(X_log, norm="l2")
elapsed_ms = (time.perf_counter() - start) * 1000

Out[17]:

Console

L2 normalization completed in 0.38ms

Row norms after normalization (first 5 docs):
  Doc 0: ||v||_2 = 1.000000
  Doc 1: ||v||_2 = 1.000000
  Doc 2: ||v||_2 = 1.000000
  Doc 3: ||v||_2 = 1.000000
  Doc 4: ||v||_2 = 1.000000

Result type: csr_matrix (stays sparse throughout)

One important detail: when you apply log scaling by modifying X_log.data directly, you're operating only on non-zero values, because .data contains only the non-zero entries. This is both efficient and correct: the zeros in the matrix represent absent terms, and they should remain zero after log scaling. If you naively applied 1 + np.log(X_full_matrix) to a dense matrix, you'd compute $1 + \log(0) = -\infty$ for all the absent terms, which is wrong. Working with the sparse .data array avoids this pitfall automatically.

Comparing the Full PipelineLink Copied

Let's time the full pipeline for the main TF variants at scale.

In[18]:

Code

import time


def benchmark_tf_variants(texts, min_df=2):
    """Time each TF variant on the given corpus."""
    results = {}

    # Raw counts baseline
    t0 = time.perf_counter()
    vec = CountVectorizer(min_df=min_df)
    X_raw = vec.fit_transform(texts)
    results["raw"] = time.perf_counter() - t0

    # Log TF
    t0 = time.perf_counter()
    X_log = X_raw.astype(float).copy()
    X_log.data = 1 + np.log(X_log.data)
    results["log"] = time.perf_counter() - t0

    # Boolean
    t0 = time.perf_counter()
    X_bool = (X_raw > 0).astype(float)
    results["boolean"] = time.perf_counter() - t0

    # L2 normalized log TF
    t0 = time.perf_counter()
    X_log_l2 = normalize(X_log.copy(), norm="l2")
    results["log+l2"] = time.perf_counter() - t0

    return results, X_raw.shape


timings, shape = benchmark_tf_variants(texts)

Out[19]:

Console

Corpus: 500 documents, 7656 terms

Timing breakdown:
  Variant                Time (ms)
--------------------------------------
  raw                        47.80
  log                         2.59
  boolean                     2.25
  log+l2                      0.50

Note: Raw count construction dominates; subsequent transformations are fast.

The raw count construction step (tokenizing text and counting terms before building the sparse matrix) dominates the total runtime. All subsequent transformations (log scaling, boolean conversion, L2 normalization) are fast because they operate only on the non-zero entries stored in .data, which is a small fraction of the full matrix. This reinforces the key practical insight: the bottleneck in a TF pipeline is almost always the initial vectorization, not the weighting scheme.