Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Quality Filtering › Perplexity Filtering

Michael BrenndoerferJanuary 10, 202610 min read

Language AI HandbookAll chapters

Part 20 of 67Data Curation

Chapter 144Quality Filtering › Perplexity Filtering

Reading section 4 of 11 · Perplexity Filtering · 10 minSection 4 of 11 · 10 min

Section 4 of 11 · 10 min read

Quality Filtering › Perplexity FilteringOne section · Sections in this chapter4 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Heuristic filters catch obvious garbage, but they cannot assess whether text is coherent and natural in a linguistic sense. A document can pass all heuristic checks and still be semantically nonsensical, machine-translated in a stilted way, or composed of grammatically valid but meaningless phrases. Perplexity filtering addresses this by scoring how well a reference language model can predict the text.

Perplexity

Perplexity measures how surprised a language model is by a sequence of text. Given a trained language model with probabilities $P$ , the perplexity of a sequence $w_1, w_2, \ldots, w_N$ is:

\text{PPL}(w_1, \ldots, w_N) = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(w_i \mid w_1, \ldots, w_{i-1})\right)

where:

$w_i$ : the $i$ -th token in the sequence
$P(w_i \mid w_1, \ldots, w_{i-1})$ : the conditional probability the model assigns to token $w_i$ given all preceding tokens
$N$ : total number of tokens in the sequence
$\exp(\cdot)$ : the exponential function, which converts the average negative log-probability back to a probability-like scale

Lower perplexity means the model assigns higher probability to the sequence. The sequence is more predictable given what the model has learned.

The key insight behind perplexity filtering is this: a language model trained on high-quality text assigns low perplexity to text that resembles its training distribution and high perplexity to text that does not. Coherent English prose gets low perplexity from a model trained on books or Wikipedia. Garbled machine translation, keyword soup, or foreign language text gets high perplexity.

This is powerful because it captures something heuristics cannot: the sequential coherence of language. A document can have perfect punctuation, normal symbol ratios, and varied vocabulary, but if the word sequences are unnatural (as happens with low-quality machine translation or automated content generation), a language model will find it surprising. The perplexity signal captures that surprise.

How Perplexity Filtering WorksLink Copied

You train or obtain a small reference language model on known-good text. Wikipedia, books corpora, or curated news articles work well. Then you score each candidate document by running it through the reference model and measuring the average perplexity. Documents above a threshold are discarded.

The reference model does not need to be large. In practice, character-level n-gram models (KenLM) or small neural models work well for this purpose. The Gopher paper and CCNet pipeline both use KenLM models trained on Wikipedia for perplexity scoring. KenLM implements a modified Kneser-Ney smoothed n-gram language model stored in a compact hash structure. It is extremely fast: scoring billions of words takes hours, not days. For a trillion-token pipeline, KenLM's throughput is essential; running a multi-billion-parameter neural model at scoring time would be prohibitively expensive.

The choice of reference corpus matters enormously. A KenLM model trained on Wikipedia learns the patterns of encyclopedic English: neutral tone, third-person perspective, factual claims, formal vocabulary. It will assign high perplexity to legal documents, poetry, informal dialogue, scientific papers, and code, not because these are low quality, but because they differ from encyclopedic style. If you want all these content types in your training corpus, you need either multiple domain-specific reference models or a permissive threshold that only removes the most extreme outliers.

Computing Perplexity with KenLMLink Copied

Let's demonstrate the concept using a simple n-gram language model, which captures the same core intuition as KenLM-based filtering.

In[5]:

Code

import math
from collections import Counter, defaultdict


class NgramLM:
    """Simple n-gram language model for perplexity scoring."""

    def __init__(self, n=3, smoothing=0.1):
        self.n = n
        self.smoothing = smoothing
        self.ngram_counts = defaultdict(Counter)
        self.vocab = set()

    def tokenize(self, text):
        return text.lower().split()

    def train(self, texts):
        for text in texts:
            tokens = ["<s>"] * (self.n - 1) + self.tokenize(text) + ["</s>"]
            self.vocab.update(tokens)
            for i in range(self.n - 1, len(tokens)):
                context = tuple(tokens[i - (self.n - 1) : i])
                word = tokens[i]
                self.ngram_counts[context][word] += 1
        self.vocab_size = len(self.vocab)

    def log_prob(self, context, word):
        context_count = sum(self.ngram_counts[context].values())
        word_count = self.ngram_counts[context][word]
        # Laplace smoothing
        return math.log(
            (word_count + self.smoothing)
            / (context_count + self.smoothing * self.vocab_size)
        )

    def perplexity(self, text):
        tokens = ["<s>"] * (self.n - 1) + self.tokenize(text) + ["</s>"]
        log_probs = []
        for i in range(self.n - 1, len(tokens)):
            context = tuple(tokens[i - (self.n - 1) : i])
            word = tokens[i]
            log_probs.append(self.log_prob(context, word))
        avg_log_prob = sum(log_probs) / len(log_probs)
        return math.exp(-avg_log_prob)


# Train on high-quality text (representative Wikipedia-style sentences)
quality_training = [
    "the researchers developed a new method for protein folding prediction",
    "climate change has accelerated glacial melting in polar regions",
    "the study examined correlations between diet and cardiovascular disease",
    "neural networks have transformed image recognition and speech processing",
    "economic growth depends on productivity gains and capital investment",
    "the algorithm achieves better performance with larger training datasets",
    "natural language processing enables computers to understand human text",
    "the experiment measured reaction times under different stimulus conditions",
    "photosynthesis converts light energy into chemical energy stored in glucose",
    "machine learning models require careful validation to avoid overfitting",
]

lm = NgramLM(n=3)
lm.train(quality_training)

In[6]:

Code

# Score candidate documents
candidates = {
    "Science news": "the scientists published their findings on neural plasticity in developing brains",
    "Coherent prose": "economic research suggests that education investment yields long term productivity gains",
    "Keyword spam": "buy cheap loans loan rates best loans cheap mortgage refinance loan calculator loans",
    "Garbled text": "the of from into systems processing language neural have been used for",
    "HTML artifact": "span class footer nav li ul div href src img alt copyright reserved",
}

Out[7]:

Console

Document                    Perplexity
----------------------------------------
Science news                      68.7
Coherent prose                    75.7
Keyword spam                      89.9
Garbled text                      69.9
HTML artifact                     89.9

The perplexity values separate the candidates along the expected lines. Coherent prose that matches the training distribution (science news, economic research sentences) gets low perplexity. Keyword spam and garbled text get high perplexity because their word sequences are unpredictable given the model's learned patterns. The HTML artifact scores high because isolated tag names and attributes have no sequential predictability from the reference model's perspective.

Notice that this works without any explicit rules about what constitutes "good" text. The n-gram model does not know that "buy cheap loans" is spam. It knows that those three words appearing together is unexpected given the contexts it has seen during training on science and technology sentences. Perplexity filtering is powerful precisely because it generalizes beyond any fixed rule set.

Perplexity Thresholds in PracticeLink Copied

The CCNet pipeline (used for XLM-R and LLaMA pretraining data) filters documents by perplexity into three buckets:

Head: Bottom 30th percentile by perplexity (highest quality text, closest to the reference distribution)
Middle: 30th to 65th percentile
Tail: Top 35th percentile by perplexity (lowest quality, often excluded entirely)

Rather than a fixed absolute threshold, CCNet uses percentile-based filtering. This is more robust because absolute perplexity values depend heavily on the reference model and the tokenization. A threshold of "reject documents with perplexity above 500" that works for English may be completely wrong for another language if you use the same model. Absolute thresholds also shift if you retrain the reference model with different hyperparameters. Percentile thresholds are stable across these variations because they are defined relative to the corpus distribution.

One important caveat is that perplexity filters can be too aggressive about domain specificity. A language model trained on Wikipedia will assign high perplexity to legal documents, poetry, and code, even though these are perfectly valid forms of text. If you want to include diverse domains in your training corpus, you need either domain-specific reference models or a more permissive threshold that only removes the truly extreme outliers. CCNet handles this partially by training separate language models for each language, but within a language the domain bias persists.

A second caveat is that perplexity filtering is circular in a subtle way. If you train your reference model on text that excludes certain dialects or styles, those dialects get high perplexity scores and are removed from the corpus. The resulting training corpus underrepresents those styles, producing a final model that performs poorly on them. The reference model's biases become the corpus's biases. This is distinct from heuristic bias (where a rule is explicitly wrong) but equally consequential.

Visualizing Perplexity DistributionsLink Copied

In[8]:

Code

import matplotlib
import matplotlib.pyplot as plt
import numpy as np

rng = np.random.default_rng(42)

# Simulate perplexity distributions for quality tiers
quality_ppl = rng.lognormal(mean=3.2, sigma=0.4, size=5000)  # high quality
medium_ppl = rng.lognormal(mean=4.2, sigma=0.5, size=5000)  # medium quality
low_ppl = rng.lognormal(mean=5.4, sigma=0.6, size=5000)  # low quality

# Clip to reasonable range for visualization
quality_ppl = np.clip(quality_ppl, 5, 800)
medium_ppl = np.clip(medium_ppl, 5, 800)
low_ppl = np.clip(low_ppl, 5, 800)

Out[9]:

Visualization

Overlapping density histograms showing perplexity distributions for three quality tiers with threshold lines. — Simulated perplexity distributions for three text quality tiers. High-quality text (solid blue) concentrates at low perplexity values, medium-quality text (dashed orange) spreads across the middle range, and low-quality text (dotted green) extends to higher values. Dashed vertical lines mark the 65th percentile thresholds used in CCNet-style filtering for each tier, illustrating why fixed absolute thresholds are less reliable than percentile-based ones.

The overlapping distributions reveal why perplexity thresholds are imperfect. The tails of the high-quality and medium-quality distributions overlap substantially. A threshold that catches most low-quality documents will inevitably discard some legitimate high-quality text. This overlap is the core precision-recall tradeoff in filtering. It is not a failure of the method: the task itself is ambiguous. Some text lies on the quality boundary, and no scalar metric will cleanly separate it. This is why production pipelines do not rely on perplexity filtering alone.

Perplexity and Tokenization SensitivityLink Copied

One practical subtlety is that perplexity scores depend on tokenization. A character-level model, a word-level model, and a BPE model trained on the same corpus will assign different perplexity values to the same document, even if all three models have similar quality. This makes it difficult to compare perplexity thresholds across different reference model implementations.

The CCNet approach of using percentile thresholds partially mitigates this: because the percentile is computed over the same corpus with the same model, relative rankings are stable even if absolute values vary. If you change the reference model (for example, updating from a 3-gram to a 5-gram KenLM model), you need to recompute the threshold percentiles on a representative sample rather than reusing old absolute cutoffs.