Content Filtering › Filter Evaluation

Building a content filter is only half the work. Evaluating whether it works, and whether it continues to work as threats evolve, is equally important and often receives less attention.

The standard evaluation framework for a binary classifier applies here, but with one necessary addition: error costs are asymmetric and depend on the harm category.

The core metrics are precision, recall, false positive rate, and AUC-ROC. Each captures a different facet of filter behavior.

Precision measures what fraction of blocked requests were harmful. If you block 100 requests and only 60 were truly harmful, your precision is 0.60:

$$\text{Precision} = \frac{TP}{TP + FP}$$

where:

• $TP$ (true positives): correctly blocked harmful requests
• $FP$ (false positives): incorrectly blocked benign requests

Low precision means high false-positive rates, which degrade user experience and erode trust.

Recall (also called sensitivity or true positive rate) measures what fraction of all harmful requests you caught. If 100 harmful requests were sent and you blocked 85, your recall is 0.85:

$$\text{Recall} = \frac{TP}{TP + FN}$$

where $FN$ (false negatives) counts harmful requests that slipped through the filter. Low recall means the filter misses too much.

False Positive Rate (FPR) measures what fraction of benign requests you incorrectly block. This is the user-experience cost of the filter:

$$\text{FPR} = \frac{FP}{FP + TN}$$

where $TN$ (true negatives) counts benign requests correctly passed through. A high FPR blocks legitimate users even when the precision and recall on harmful content look good.

F1 Score is the harmonic mean of precision and recall, giving a single balanced summary:

$$F_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$

The harmonic mean is used (rather than the arithmetic mean) because it penalizes extreme imbalance: a system with precision 1.0 and recall 0.0 would get a misleadingly high arithmetic mean of 0.5, but an F1 score of 0.0, correctly reflecting that it catches nothing.

AUC-ROC: The area under the receiver operating characteristic curve, which measures classifier quality across all possible thresholds. An AUC of 0.5 means random guessing; 1.0 means perfect.

For safety applications, it is common to report performance at a fixed false positive rate: "What is our recall when we hold false positive rate at 1%?" This reflects the operational constraint that you cannot block too many legitimate users.

Another reporting convention is to specify recall at a fixed precision, for example: "At 90% precision, what fraction of harmful requests do we catch?" This framing is useful when the cost of false positives is explicitly bounded by business requirements, and you want to know how much safety protection is achievable within that constraint.

It is worth understanding the mathematical relationship between these metrics and Bayes' theorem. The precision of a filter is not a fixed property of the classifier; it depends on the prevalence of harmful content in your traffic. If $h$ is the prior probability that any given request is harmful, $TPR$ is the true positive rate (recall), and $FPR$ is the false positive rate, then the precision of the filter is:

$$\text{Precision} = \frac{TPR \cdot h}{TPR \cdot h + FPR \cdot (1 - h)}$$

This formula has a powerful practical implication: when harmful content is rare ($h$ is small), the denominator is dominated by the $FPR \cdot (1 - h)$ term, and precision collapses even for classifiers with high AUC. The visualization later in this section illustrates this effect concretely.

Evaluating content filters is harder than evaluating most classifiers because of three challenges.

Class imbalance: Harmful content is rare. In a typical API deployment, well under 1% of requests are harmful. When you evaluate your filter on a random sample of production traffic, almost all examples are benign. To get a meaningful estimate of recall, you need an evaluation set that deliberately oversamples harmful examples. This requires either careful collection of known-harmful content or synthetic generation of adversarial examples.

Distribution shift: The distribution of harmful content changes over time as adversaries discover new attack patterns and as the user population changes. A filter trained on data from six months ago may miss newly emerging jailbreak patterns. Evaluation on historical test data tells you how well the filter worked in the past, not how well it works now.

Adversarial examples: A sophisticated attacker will not phrase their harmful request in the same way as the examples in your training set. They will use paraphrases, coded language, or multi-turn strategies designed to evade the filter. Evaluation against non-adversarial examples overestimates the filter's effectiveness against determined adversaries. This is why red team evaluation is an essential complement to standard classification metrics.

These three challenges compound each other in an important way. A filter evaluated on a balanced historical dataset with straightforward harmful examples will look much better than it performs in production against a sparse stream of evolving adversarial content. This gap between evaluation metrics and operational performance is one of the most persistent problems in content filtering practice. Closing it requires ongoing adversarial probing, diverse evaluation sets, and a monitoring infrastructure that surfaces real failures as they occur.

A red team evaluation of a content filter involves testers who deliberately try to elicit harmful outputs while evading the filters. Unlike standard evaluation (which tests the filter on a fixed dataset), red team evaluation is adversarial: the testers adapt their strategies based on what the filter blocks and what it misses.

Red team evaluation should cover:

• Direct attempts: Straightforward requests for harmful content, to establish baseline recall
• Paraphrase attacks: Semantically equivalent requests phrased differently
• Encoding attacks: Character substitutions, base64 encoding, or foreign language
• Context manipulation: Embedding harmful requests in fictional contexts, roleplay, or hypothetical framings
• Jailbreak templates: Known prompt patterns that have historically bypassed model safety training (as covered in the Jailbreaking chapter)

The goal is to find the filter's weaknesses before adversaries do. Red team findings should feed directly into training data collection for the next version of the filter.

Red team evaluation is most effective when the testers have deep knowledge of the harm categories they are probing, access to a diverse range of bypass techniques, and freedom to experiment without constraints. Many organizations engage external red teams specifically because internal teams may have blind spots shaped by the same assumptions that were baked into the filter design. External teams bring fresh adversarial perspectives and are not anchored to the mental models that shaped the original training data.

The outputs of red team evaluation should include a list of successful bypasses and a characterization of the filter's failure modes. Are the failures concentrated in specific harm categories? Do they tend to occur at specific text lengths or conversation depths? Are they mostly paraphrase attacks or encoding attacks? This characterization guides prioritization of improvements and helps identify whether the failures are better addressed through more training data, a different architecture, or a revised taxonomy.

In production, evaluation is not a one-time event. You need ongoing monitoring to detect when filter performance degrades. Key monitoring signals include:

• Block rate over time: A sudden spike in the fraction of blocked requests suggests either a new attack pattern or a filter that has become too aggressive. A sudden drop may indicate filter degradation.
• False positive complaints: User reports of incorrectly blocked content are a direct signal of false positive rate changes.
• Sampling and human review: Periodically sample blocked requests and have human reviewers assess whether the blocks were correct.
• Adversarial probe sets: Maintain a fixed set of known-harmful probe requests and run them against the filter regularly. If the pass-through rate on these probes increases, the filter is degrading.

Drift detection is particularly important for ML-based filters. A model deployed six months ago was trained on data that may no longer represent current threats. Regularly retraining on recent production data, including new red team findings, keeps the filter current.

Monitoring should be treated as a first-class engineering concern, not an afterthought. A filter with no monitoring is effectively flying blind: you have no way to know whether it is working until a high-profile failure surfaces in the press or through user complaints. Investing in monitoring infrastructure early makes it possible to catch degradation quietly and fix it proactively rather than reactively.

---