Content Filtering: Classification, Rules, and Evaluation

Every language model deployment faces a fundamental challenge: users will send inputs you did not anticipate, and the model will sometimes produce outputs you would never intentionally allow. Content filtering is the systematic layer of detection and blocking that sits around language models to prevent harmful inputs from reaching them and harmful outputs from reaching users. It is one of the most practically important tools in the safety engineer's toolkit, and understanding how it works, where it succeeds, and where it quietly fails will make you a more effective practitioner.

In the previous chapters of this part, you learned how safety risks emerge from language models (Safety Risks), how red teams probe for vulnerabilities (Red Teaming), and how jailbreaks and prompt injections try to circumvent model safeguards (Jailbreaking, Prompt Injection). Content filtering takes a different stance: rather than trying to make the model itself safe, it adds external detection layers that intercept problematic content before or after the model runs. Think of it as airport security rather than pilot training. Even if your pilots are well-trained, you still screen passengers at the gate.

This chapter covers the two main families of filtering approaches (classification-based and rule-based), where filters are placed in the system architecture, and how to evaluate whether a filter is doing its job. By the end, you will understand the full design space well enough to design and assess content filters for real deployments. You will also learn why class imbalance fundamentally distorts precision in production environments, how adversaries probe and defeat filters over time, and what it takes to build a filtering system that ages well.

Content filtering refers to any automated mechanism that examines text and makes a binary or multi-label decision about whether that text is safe to process or return. The key word is "automated." Filters run on every single request, so they must be fast and cheap. Human review, while essential for evaluating filter quality, cannot happen in the critical path of a live system.

Filtering is not the same as alignment. Alignment training (covered in earlier chapters on RLHF and constitutional AI) modifies the model's weights so that it is less likely to generate harmful content in the first place. Filtering adds a separate, modular detection layer. The two are complementary: alignment reduces the base rate of harmful outputs, while filtering catches the cases that slip through. In practice, production systems use both.

Filtering is also not the same as guardrails, which you will explore in the next chapter. Guardrails is a broader term that includes more complex policy enforcement, multi-turn safety logic, and structured output validation. Content filters are typically the simpler, faster, lower-latency components that do the initial screening.

It is also worth clarifying what filtering cannot do. A filter cannot guarantee zero harmful outputs. No automated system can perfectly identify all possible harmful content, because harm depends on context, changes over time, and has contested definitions. Harm categories shift with social norms, legal systems, and deployment contexts. What is universally prohibited (child sexual abuse material, detailed synthesis instructions for weapons of mass destruction) sits alongside categories that are context-dependent (violence in a war history course vs. a children's app, drug information in a harm-reduction service vs. a general chat application). Content filters encode an approximation of these judgments at a given moment in time, and that approximation is always incomplete.

Understanding these boundaries matters because it shapes how you architect systems. If you treat the filter as the single safeguard, you will be surprised when it fails. If you treat it as one layer among many, its limitations become manageable. Filters are most valuable when they reduce the throughput of harmful content to levels that other layers (alignment training, human review, account controls) can handle, not when they are expected to eliminate harm entirely on their own.

The most widely deployed approach to content filtering is to treat it as a supervised classification problem. You define a set of harm categories, collect labeled examples, and train a model that predicts which (if any) categories apply to a given text.

Before you can train a classifier, you need to decide what you are classifying. This requires a harm taxonomy: a structured list of categories that represent the kinds of content you want to block.

Different organizations have settled on different taxonomies, but common categories include:

• Hate speech: Content that degrades or dehumanizes individuals or groups based on protected characteristics such as race, religion, gender, or sexual orientation.
• Harassment: Targeted threats, insults, or intimidation directed at specific people.
• Sexual content: Explicit sexual material, which may have subcategories for adult-only content versus content involving minors (which is universally prohibited).
• Violence: Graphic descriptions of physical harm, instructions for violent acts, or glorification of violence.
• Self-harm: Content that promotes, instructs, or glorifies suicide or self-injury.
• Dangerous information: Instructions for weapons, drugs, malware, or other content that enables real-world harm.
• Spam and manipulation: Scam content, phishing attempts, or coordinated inauthentic behavior.

The boundary between categories is often fuzzy. A piece of text might be simultaneously violent and hateful. A description of drug effects might be educational in one context and dangerous in another. For this reason, most production classifiers are trained as multi-label classifiers rather than single-label ones: a single input can trigger multiple categories simultaneously.

OpenAI's moderation API, for example, uses categories like hate, hate/threatening, harassment, harassment/threatening, self-harm, self-harm/intent, self-harm/instructions, sexual, sexual/minors, violence, and violence/graphic. The hierarchical structure (e.g., self-harm/intent is more specific than self-harm) allows downstream systems to apply different thresholds to different severity levels.

Designing a good harm taxonomy is harder than it looks. Categories must be specific enough to be trainable (you need examples that clearly fall in or out of each category) but broad enough to cover the range of harmful content in each semantic family. If a category is too narrow, adversaries can easily rephrase their way around it. If it is too broad, the classifier will block too much legitimate content. The process of building a taxonomy often reveals implicit assumptions about what counts as harmful, and surfacing those assumptions explicitly is one of the most valuable outcomes of the taxonomy-building exercise.

Taxonomies also need to be revisited regularly. Harm categories that were well-defined two years ago may need subdivision as new attack patterns emerge, or may need merger as the operational cost of maintaining separate classifiers becomes prohibitive. Organizations that treat their taxonomy as a fixed artifact rather than a living document tend to find that their filters gradually drift out of alignment with actual harm patterns in production.

A classification filter is typically a fine-tuned language model, though simpler text classifiers are used when latency is critical. The training pipeline involves three main components.

Data collection is the hardest part. You need examples of both harmful and benign content that reflect the actual distribution you will encounter in production. Harmful examples come from multiple sources: red team exercises (where human testers deliberately generate boundary cases), human-written datasets like the HateXplain corpus or ToxiGen, synthetic generation (where you prompt a capable model to produce borderline examples and then filter them), and production logs from real deployments (with appropriate privacy protections). The data must be carefully balanced, because harmful content is rare by definition. A model trained on a corpus where only 0.1% of examples are harmful will learn to predict "safe" for everything and achieve 99.9% accuracy while being completely useless.

The diversity of training data deserves special emphasis. A dataset that captures only the most obvious, explicit forms of harmful content will produce a classifier that is easily fooled by subtle paraphrases, euphemisms, or culturally specific expressions. Red team annotators with adversarial mindsets, subject matter experts in specific harm domains (e.g., child safety experts for CSAM-related categories, counter-terrorism analysts for violent extremism), and crowdsourced annotation from diverse populations all contribute data characteristics that no single source can provide on its own.

Labeling is the process of assigning harm categories to examples. Because the category boundaries are fuzzy and sometimes contested, labeling requires clear guidelines, multiple annotators per example, and inter-annotator agreement metrics to catch cases where annotators disagree. Disagreement is often informative: a high inter-annotator disagreement rate on a category suggests that the category is ambiguously defined and needs refinement.

Inter-annotator agreement is commonly measured using Cohen's kappa coefficient $\kappa$, which corrects for the probability of chance agreement:

where:

• $p_o$: the observed proportion of cases where annotators agree
• $p_e$: the expected proportion of cases where annotators agree by chance, computed from the marginal distributions of each annotator's labels

A $\kappa$ value near 1.0 indicates near-perfect agreement; a value near 0.0 indicates agreement no better than random chance; negative values indicate systematic disagreement. For safety classification tasks, a $\kappa$ below 0.6 on a given category is typically a signal to revisit the labeling guidelines for that category before using the data for training.

Model architecture for filtering classifiers typically follows one of three patterns. A bidirectional encoder like BERT or RoBERTa is fine-tuned on the classification task. The encoder reads the full text, produces a [CLS] token embedding, and a classification head on top predicts the harm category probabilities. Alternatively, a full autoregressive model (GPT-class) can be prompted or fine-tuned to output category labels. Finally, for embedding-based approaches, you can use a pretrained text encoder to project texts into a vector space and then apply a lightweight classifier (logistic regression, SVM) on top of the embeddings.

The advantage of encoder-based classifiers is speed and simplicity. A fine-tuned BERT model running on GPU can classify tens of thousands of texts per second. The advantage of LLM-based classifiers is nuance: large models better understand context and sarcasm across a multi-turn conversation. Many systems use both: a fast classifier for the common case and a more expensive LLM-based check for high-stakes or uncertain cases.

A third option that is gaining traction is using a large LLM as a zero-shot or few-shot judge, where you prompt the model with the harm taxonomy and a few labeled examples and ask it to evaluate each new input. This approach requires no fine-tuning and can be updated by modifying the prompt rather than retraining, which makes it very agile for responding to new threat patterns. The cost is high: LLM inference is orders of magnitude more expensive than a small fine-tuned encoder, and running an LLM judge on every request is only practical at low request volumes or when the cost of false negatives is extremely high. In practice, the LLM-as-judge pattern is often deployed asynchronously for quality monitoring rather than in the real-time request path.

A classification model outputs probabilities, not hard labels. You must choose a threshold above which a probability triggers a block. This choice is a policy decision, not a technical one.

Setting the threshold involves a fundamental tradeoff between two types of errors:

• False positives (blocking benign content): These frustrate users, degrade the product experience, and, in some cases, cause real harm. A medical information service that blocks all mentions of "overdose" will fail users seeking harm-reduction information. A creative writing platform that blocks all violent content will frustrate novelists. False positives erode user trust.
• False negatives (allowing harmful content through): These represent the filter's failure to catch what it was designed to catch. The consequences depend on the category and use case: allowing one piece of hate speech in a consumer product is bad; allowing a bioweapon synthesis request in a research API is catastrophic.

The right threshold depends on your use case, your user population, and your harm model. A children's education platform should set a very low threshold for sexual content (high sensitivity) and accept a higher false positive rate. An adult content platform might set a very high threshold for most harm categories and only strictly enforce legal violations. Most platforms adjust thresholds per category rather than applying a single global threshold.

It is useful to frame threshold selection as an optimization problem. Define the business cost $C_{FP}$ of a false positive and the business cost $C_{FN}$ of a false negative for each harm category. The optimal threshold $\tau^*$ minimizes expected total cost:

where $\hat{y}(\tau) = \mathbf{1}[p(y=1|x) \geq \tau]$ is the predicted label at threshold $\tau$. In practice, $C_{FP}$ and $C_{FN}$ are rarely quantified precisely, but the framing is useful: it forces you to articulate the relative cost of each error type and makes the policy nature of threshold selection explicit. A threshold of 0.5 is not a neutral default; it implicitly assumes the cost of a false positive equals the cost of a false negative, which is almost never true in safety applications.

Text that is harmful in one context may be benign in another. The phrase "how do I kill this process?" is dangerous in a conversation about biological weapons but completely normal in a software debugging context. Contextual classifiers address this by including conversation history, system prompt, and metadata (user role, application type, API tier) as input to the classification model rather than relying on the single message.

This is one of the key advantages of LLM-based classifiers over simple text classifiers. When you feed a whole conversation thread to a capable language model with the prompt "Is this exchange harmful?", the model can use the full context to make a more context-sensitive judgment. Simple bag-of-words or keyword-based classifiers cannot.

The downside is latency and cost. Running a full LLM classifier on every message of a high-volume API adds significant overhead. In practice, many systems use a tiered approach: a fast low-latency classifier runs on every request, and a slow high-quality classifier runs only on requests that the fast classifier flagged as uncertain (probability between 0.3 and 0.7, for example) or on a random sample for quality monitoring.

Context windows also create privacy risks. If you feed the full conversation history to a classifier API, you are transmitting potentially sensitive user content to the classification service. In some jurisdictions, this raises regulatory compliance questions. Architectures that keep classification in-house (running the classifier on your own infrastructure) avoid this concern but require more operational overhead than calling a third-party API.

Metadata signals can help contextual classification even when full conversation history is not available. Knowing that a request comes from an API tier that requires identity verification, from an account that has been active for three years, or from a system prompt that authorizes adult content significantly shifts the prior probability of harm. Some filter architectures encode these signals as additional input features to the classifier, rather than requiring the full context to be re-analyzed linguistically.

It is worth being precise about what "multi-label" means here, because it differs in important ways from ordinary multi-class classification. In a standard multi-class classifier, each input belongs to exactly one class: a digit image is a 3 or a 7, not both. In multi-label classification, each input can belong to any subset of the available categories simultaneously. A message can be both hateful and threatening; a piece of text can contain both sexual content and violence.

This distinction has practical implications for how you build and calibrate the classifier. In a multi-class setting, you apply a softmax activation over the output logits to produce a probability distribution that sums to 1.0 across all classes. In a multi-label setting, you apply a sigmoid activation to each logit independently, producing probabilities that are each in the range $(0, 1)$ and do not sum to any fixed total. Each category has its own independent probability estimate. You then apply a per-category threshold to decide which labels to assign.

The sigmoid function applied to logit $z_k$ for category $k$ is:

Each $p_k$ is treated as an independent binary probability. The label prediction for category $k$ at threshold $\tau_k$ is then simply:

Because the thresholds $\tau_k$ are per-category, you tune them independently based on the operational requirements for each harm type. The threshold for violence/graphic in a general consumer app might be 0.3, while the threshold for hate/non-threatening in a research API might be 0.8.

The independence assumption in multi-label classification is a simplification. Categories are not truly independent: a text that scores high for "threat" is more likely to also score high for "harassment" than a random text. Some systems model these correlations explicitly using label-correlation architectures or by chaining classifiers (first predict the primary category, then predict secondary categories conditioned on the primary). In practice, the simple independent sigmoid approach works well enough for most use cases, because the categories are correlated in ways the model learns during training regardless.

Before machine learning was applied to content moderation, rule-based filtering was the dominant approach, and it remains valuable today for specific use cases where precision is critical and the patterns are well-defined.

The simplest rule-based filter is a blocklist: a list of words, phrases, or patterns that trigger an immediate block. If any item in the list appears in the text, the text is blocked. No machine learning required.

Blocklists have real advantages. They are interpretable (you can audit exactly what triggers a block), they are fast (string matching is extremely cheap compared to model inference), and they are deterministic (you know exactly when they will fire). For categories where the harmful content is reliably expressed through specific terms, such as illegal product names, known scam phrases, or regulatory keywords, blocklists can achieve near-perfect precision on their specific patterns.

The limitations are equally well-known. Blocklists are brittle: they block the exact strings they list, but miss paraphrases, misspellings, and synonyms. A blocklist for the word "kill" will also block "skill", "killer whale", "kill switch in the factory process". This over-blocking problem is called the Scunthorpe problem, after the English town whose name triggered early internet profanity filters. Conversely, a user motivated to bypass a blocklist can do so trivially through character substitution, spacing, or language switching.

For this reason, blocklists are almost never used as a standalone filter in modern systems. They function as a fast first-pass layer or as a precision tool for high-severity known patterns, supplemented by classifier-based detection.

The maintenance burden of blocklists is also often underestimated. A blocklist that covers the known threats as of launch day will be incomplete three months later. New slang terms emerge, adversaries discover gaps, and user populations shift. Keeping a blocklist current requires ongoing human monitoring of production traffic, which is expensive and easy to deprioritize. Organizations that invest in automated ML-based detection can often afford to let blocklists age more gracefully, because the classifier provides a backstop for patterns the blocklist misses.

Regular expressions (covered in detail in Part I: Text as Data of this book, particularly the Regular Expressions chapter) give you more expressive rule patterns than simple string matching. You can write patterns that match structural properties of harmful content rather than just specific words.

Common use cases for regex-based filters include:

• Personal identifiable information (PII) detection: Patterns for credit card numbers, Social Security numbers, email addresses, and phone numbers.
• URL pattern filtering: Blocking known malicious domains or filtering all external links in certain contexts.
• Structured content detection: Detecting code patterns associated with SQL injection or command injection.
• Format violations: Ensuring outputs conform to required formats (no raw HTML in a plaintext response, for example).

The advantage of regex filters over ML classifiers for these cases is precision. A regex for a 16-digit credit card number will match credit card numbers reliably without false positives on other numeric strings. A classifier trained to detect PII might flag "he is 45 years old" as sensitive, because it has learned to associate numbers with PII.

The limitation is that regex is brittle to slight variations and cannot generalize to new patterns it was not written to match. A credit card regex written for 16-digit Visa and Mastercard numbers will miss 15-digit American Express numbers unless you remember to add that pattern. A phone number regex that handles US formats will miss international formats unless each country's format is explicitly encoded. Maintaining broad coverage requires dedicated engineering attention and regular updates.

Despite these limitations, regex-based filters are often the right choice for PII detection and format enforcement, because the cost of a false negative (leaking a user's credit card number or Social Security number) is high, the patterns are relatively stable, and the precision requirements are strict. ML classifiers trained on noisy internet data tend to have higher false positive rates on structured PII patterns than a well-written regex.

Beyond keywords and regexes, some systems use structured rule engines that combine multiple conditions. A rule might say: "Block if the message mentions substance X AND the user's account is less than 30 days old AND the application is not in the approved-use list." This kind of multi-factor rule captures business logic that is difficult to encode in a classifier: the harm is not a property of the text alone but of the text in combination with context signals.

Rule systems are common in fraud detection and content policy enforcement at scale, where the policy logic is complex but interpretable and needs to be auditable. The downside is maintenance burden: rule systems require ongoing engineering work to update as threats evolve.

The advantage of structured rule systems over purely ML-based approaches is their auditability and controllability. When a regulator or legal team asks why a particular request was blocked, a rule system produces a clear, human-readable audit trail. A neural classifier produces a probability score that cannot easily be traced to specific text features. For organizations subject to regulatory requirements around automated decision-making (such as GDPR's right to explanation), this auditability can be a significant operational advantage.

Some organizations combine rule systems with ML classifiers in a cascade: the rule system handles the clear-cut, high-severity cases where policy is explicit and unambiguous, and the ML classifier handles the ambiguous middle ground. This gives you the best of both worlds: fast, interpretable, high-precision blocks for known patterns, and flexible, context-sensitive judgment for the cases where rules alone are insufficient.

Where in the request-response pipeline you place a content filter matters as much as how you build it. A filter can sit at four distinct points: before the user input reaches the model, within the model's generation process, after the model produces output but before it is returned, or asynchronously after the response is already delivered.

Input filters screen user messages before they reach the model. If the input is flagged as harmful, the system can refuse to process it, respond with a canned refusal message, or route the request to a human reviewer.

Input filtering is the most common placement and has clear advantages. It prevents the model from being exposed to prompt injection payloads, jailbreak attempts, and requests for dangerous information. It is also computationally cheaper than output filtering, because you only need to run the filter once per request.

The limitation of input-only filtering is that it leaves output quality entirely to the model. Even with a clean input, the model might produce harmful outputs through hallucination, mode collapse, or unexpected generalization. A model asked to write a fictional story might produce deeply disturbing content even from an innocuous prompt. An adversary who understands that only input is filtered can craft inputs that are superficially benign but steer the model toward harmful completions through indirect means.

Input filters also face the challenge of ambiguity. A question about medication dosages could come from a pharmacist, a patient, a caregiver, or someone planning self-harm. A question about firearm modifications could come from a gunsmith, a competitive shooter, or someone planning violence. Input filters that are too aggressive block legitimate users; input filters that are too permissive pass requests that produce harmful outputs. The right calibration depends on knowing your user population, which is often difficult.

Output filters screen the model's response before returning it to the user. If the output is flagged, the system can either suppress the response entirely (returning a refusal message) or attempt to regenerate a safer alternative.

Output filtering is essential for catching cases that input filtering misses. A model might produce harmful content in response to a seemingly benign request ("Write a story about a character who explains how to..."), and an input filter would never detect this. Output filters catch the actual harmful content regardless of how it was elicited.

The main cost of output filtering is latency and resource usage. You must run the entire model generation before you can evaluate the output. If you detect a problem and need to regenerate, you have doubled (or more) your latency and compute cost. For streaming responses (where tokens are returned to the user as they are generated), this creates additional complexity: you cannot filter a response that is still being generated.

Some systems address the streaming problem with a speculative approach: run the output filter on the first 50-100 tokens as they arrive, and if the early tokens look dangerous, interrupt the stream and send a refusal. This adds slight latency to the first tokens but avoids the full cost of generating a complete harmful response before filtering. The effectiveness of this approach depends on how early in the response the harmful content typically appears; if a model front-loads its refusals or preambles before the harmful content, early-token filtering may miss it.

Another approach is to buffer the stream entirely on the server side before forwarding it to the client, applying the output filter to the complete response before any tokens reach the user. This sacrifices the perceived interactivity of streaming but keeps the filtering logic simple. Which approach you choose depends on how important the streaming experience is to your users and how latency-sensitive your filtering requirements are. Consumer chat applications tend to prioritize streaming interactivity; high-stakes enterprise applications may prefer the safety of buffered output filtering.

Output filtering also creates a decision about what to do with filtered responses. You can replace the harmful response with a canned refusal ("I can't help with that request."), attempt to regenerate a safer alternative, or return nothing at all. Each choice has tradeoffs. Canned refusals are fast and predictable but may frustrate users with legitimate requests. Regeneration addresses the user's underlying need but doubles latency and may still fail if the model keeps producing similar outputs. Returning nothing at all maximizes safety but is the worst experience for users whose requests were blocked incorrectly.

Production systems typically use both input and output filters. Input filtering catches the obvious harmful requests before wasting compute on model inference. Output filtering catches the subtler cases where the model produces harmful content despite a clean input. The combination provides defense in depth: an attacker must craft a prompt that bypasses both the input filter and the model's own safety training, and produce an output that also bypasses the output filter.

The value of bidirectional filtering compounds with the diversity of the two filters. If both input and output filters use the same underlying model, they will likely share the same blind spots, and an adversary who defeats one will likely defeat both. Using architecturally distinct filters (e.g., a fast keyword classifier at input and a fine-tuned transformer at output, or an embedding-based classifier at input and an LLM judge at output) means that a bypass technique targeting one filter's specific weaknesses is less likely to defeat the other.

Not all filtering needs to happen in real time. Asynchronous filters run after the response is already delivered to the user, usually for monitoring and retrospective analysis. They can flag conversations for human review, update training data for filter improvement, or trigger account actions (warnings, suspensions) based on patterns of behavior over time.

Asynchronous filters can afford to be much more expensive and accurate than real-time filters, because they are not in the critical latency path. You can run a large LLM-based judge, apply multiple classifiers in parallel, or perform complex multi-turn analysis on entire conversation histories. The tradeoff is that harmful content may have already reached the user before the flag is raised. For most categories this is acceptable (retrospective action is still valuable), but for categories where real-time prevention is critical (e.g., CSAM, bioweapon synthesis instructions), asynchronous-only filtering is insufficient.

Asynchronous filters are also the most practical mechanism for maintaining and improving your real-time filters. By continuously reviewing a sample of production traffic with high-quality classifiers, you build a feedback loop: edge cases found by the async filter become training data for the next version of the real-time filter. This feedback loop is one of the most important architectural features of a mature content filtering system.

The choice of filter placement depends on the threat model, the latency budget, and the acceptable false-positive rate. The key considerations are:

• Input filters are fast and cheap, but they miss output-side failures
• Output filters provide broad coverage but add latency and complexity for streaming
• Both together provide defense in depth at higher total cost
• Async filters are expensive and accurate, but reactive rather than preventive

Let's build a classification-based content filter from scratch to see how these concepts work in practice. We will use scikit-learn to train a logistic regression classifier on TF-IDF features from a synthetic toxicity dataset. This keeps the demonstration self-contained and executable without requiring a GPU or HuggingFace model download.

Notice the clear tradeoff: lowering the threshold increases recall (we catch more harmful content) but also raises the false positive rate (more benign content is blocked). Threshold 0.3 catches nearly all harmful content at the cost of blocking some safe requests; threshold 0.7 produces almost no false positives but misses a significant fraction of harmful inputs. The optimal threshold depends on which type of error is more costly for your application.

The key parameters for a classification-based content filter are:

• model architecture: TF-IDF + logistic regression for maximum speed and interpretability; fine-tuned transformer encoders (BERT, RoBERTa) for better accuracy on context-dependent content; LLMs for contextual multi-turn reasoning.
• harm taxonomy: The set of categories the classifier detects. More categories require more labeled training data per category.
• threshold: The probability cutoff controlling precision vs. recall. Should be tuned per category based on the business cost of each error type.
• ngram_range (for TF-IDF): Including bigrams and trigrams captures multi-word phrases that single-word features miss.
• context window: Whether the classifier sees only the current message or the full conversation history. Wider context reduces false positives and negatives on context-dependent content.

Let's also implement a rule-based filter for detecting and redacting personally identifiable information (PII) in model outputs. This is a common practical need where the regularity of the patterns makes regex a better choice than ML.

The rule-based PII filter reliably catches structured patterns. Notice how combining both ML classification and regex-based rule filters covers complementary failure modes: the ML classifier handles context-dependent semantic content (hate, threats, harmful instructions) while regex handles structured format violations (PII, card numbers, credentials) with higher precision.

Building a content filter is only half the work. Evaluating whether it works, and whether it continues to work as threats evolve, is equally important and often receives less attention.

The standard evaluation framework for a binary classifier applies here, but with one necessary addition: error costs are asymmetric and depend on the harm category.

The core metrics are precision, recall, false positive rate, and AUC-ROC. Each captures a different facet of filter behavior.

Precision measures what fraction of blocked requests were harmful. If you block 100 requests and only 60 were truly harmful, your precision is 0.60:

where:

• $TP$ (true positives): correctly blocked harmful requests
• $FP$ (false positives): incorrectly blocked benign requests

Low precision means high false-positive rates, which degrade user experience and erode trust.

Recall (also called sensitivity or true positive rate) measures what fraction of all harmful requests you caught. If 100 harmful requests were sent and you blocked 85, your recall is 0.85:

where $FN$ (false negatives) counts harmful requests that slipped through the filter. Low recall means the filter misses too much.

False Positive Rate (FPR) measures what fraction of benign requests you incorrectly block. This is the user-experience cost of the filter:

where $TN$ (true negatives) counts benign requests correctly passed through. A high FPR blocks legitimate users even when the precision and recall on harmful content look good.

F1 Score is the harmonic mean of precision and recall, giving a single balanced summary:

The harmonic mean is used (rather than the arithmetic mean) because it penalizes extreme imbalance: a system with precision 1.0 and recall 0.0 would get a misleadingly high arithmetic mean of 0.5, but an F1 score of 0.0, correctly reflecting that it catches nothing.

AUC-ROC: The area under the receiver operating characteristic curve, which measures classifier quality across all possible thresholds. An AUC of 0.5 means random guessing; 1.0 means perfect.

For safety applications, it is common to report performance at a fixed false positive rate: "What is our recall when we hold false positive rate at 1%?" This reflects the operational constraint that you cannot block too many legitimate users.

Another reporting convention is to specify recall at a fixed precision, for example: "At 90% precision, what fraction of harmful requests do we catch?" This framing is useful when the cost of false positives is explicitly bounded by business requirements, and you want to know how much safety protection is achievable within that constraint.

It is worth understanding the mathematical relationship between these metrics and Bayes' theorem. The precision of a filter is not a fixed property of the classifier; it depends on the prevalence of harmful content in your traffic. If $h$ is the prior probability that any given request is harmful, $TPR$ is the true positive rate (recall), and $FPR$ is the false positive rate, then the precision of the filter is:

This formula has a powerful practical implication: when harmful content is rare ($h$ is small), the denominator is dominated by the $FPR \cdot (1 - h)$ term, and precision collapses even for classifiers with high AUC. The visualization later in this section illustrates this effect concretely.

Evaluating content filters is harder than evaluating most classifiers because of three challenges.

Class imbalance: Harmful content is rare. In a typical API deployment, well under 1% of requests are harmful. When you evaluate your filter on a random sample of production traffic, almost all examples are benign. To get a meaningful estimate of recall, you need an evaluation set that deliberately oversamples harmful examples. This requires either careful collection of known-harmful content or synthetic generation of adversarial examples.

Distribution shift: The distribution of harmful content changes over time as adversaries discover new attack patterns and as the user population changes. A filter trained on data from six months ago may miss newly emerging jailbreak patterns. Evaluation on historical test data tells you how well the filter worked in the past, not how well it works now.

Adversarial examples: A sophisticated attacker will not phrase their harmful request in the same way as the examples in your training set. They will use paraphrases, coded language, or multi-turn strategies designed to evade the filter. Evaluation against non-adversarial examples overestimates the filter's effectiveness against determined adversaries. This is why red team evaluation is an essential complement to standard classification metrics.

These three challenges compound each other in an important way. A filter evaluated on a balanced historical dataset with straightforward harmful examples will look much better than it performs in production against a sparse stream of evolving adversarial content. This gap between evaluation metrics and operational performance is one of the most persistent problems in content filtering practice. Closing it requires ongoing adversarial probing, diverse evaluation sets, and a monitoring infrastructure that surfaces real failures as they occur.

A red team evaluation of a content filter involves testers who deliberately try to elicit harmful outputs while evading the filters. Unlike standard evaluation (which tests the filter on a fixed dataset), red team evaluation is adversarial: the testers adapt their strategies based on what the filter blocks and what it misses.

Red team evaluation should cover:

• Direct attempts: Straightforward requests for harmful content, to establish baseline recall
• Paraphrase attacks: Semantically equivalent requests phrased differently
• Encoding attacks: Character substitutions, base64 encoding, or foreign language
• Context manipulation: Embedding harmful requests in fictional contexts, roleplay, or hypothetical framings
• Jailbreak templates: Known prompt patterns that have historically bypassed model safety training (as covered in the Jailbreaking chapter)

The goal is to find the filter's weaknesses before adversaries do. Red team findings should feed directly into training data collection for the next version of the filter.

Red team evaluation is most effective when the testers have deep knowledge of the harm categories they are probing, access to a diverse range of bypass techniques, and freedom to experiment without constraints. Many organizations engage external red teams specifically because internal teams may have blind spots shaped by the same assumptions that were baked into the filter design. External teams bring fresh adversarial perspectives and are not anchored to the mental models that shaped the original training data.

The outputs of red team evaluation should include a list of successful bypasses and a characterization of the filter's failure modes. Are the failures concentrated in specific harm categories? Do they tend to occur at specific text lengths or conversation depths? Are they mostly paraphrase attacks or encoding attacks? This characterization guides prioritization of improvements and helps identify whether the failures are better addressed through more training data, a different architecture, or a revised taxonomy.

In production, evaluation is not a one-time event. You need ongoing monitoring to detect when filter performance degrades. Key monitoring signals include:

• Block rate over time: A sudden spike in the fraction of blocked requests suggests either a new attack pattern or a filter that has become too aggressive. A sudden drop may indicate filter degradation.
• False positive complaints: User reports of incorrectly blocked content are a direct signal of false positive rate changes.
• Sampling and human review: Periodically sample blocked requests and have human reviewers assess whether the blocks were correct.
• Adversarial probe sets: Maintain a fixed set of known-harmful probe requests and run them against the filter regularly. If the pass-through rate on these probes increases, the filter is degrading.

Drift detection is particularly important for ML-based filters. A model deployed six months ago was trained on data that may no longer represent current threats. Regularly retraining on recent production data, including new red team findings, keeps the filter current.

Monitoring should be treated as a first-class engineering concern, not an afterthought. A filter with no monitoring is effectively flying blind: you have no way to know whether it is working until a high-profile failure surfaces in the press or through user complaints. Investing in monitoring infrastructure early makes it possible to catch degradation quietly and fix it proactively rather than reactively.

This heatmap reveals a critical insight for filter evaluation: precision is not a fixed property of the classifier. It depends jointly on classifier quality (AUC), the selected operating point, and the prevalence of harmful content in your production traffic. Under the symmetric Gaussian operating-point assumption used here, even a strong classifier with AUC 0.95 achieves only about 7% precision when harmful requests make up just 1% of traffic. This means roughly 93% of blocked requests are false positives, not because the classifier is bad, but because the class imbalance is so extreme that even a small FPR generates many more false alarms than true alarms. This effect, known as the base rate fallacy, is why you should always evaluate filter performance on adversarially collected datasets with controlled class balance, not just on random production samples where precision numbers will look deceptively low.

One aspect of content filtering that deserves its own discussion is the adversarial dynamic: filters do not operate in a static environment. Adversaries actively probe systems to find bypass techniques, and the available filtering methods evolve continuously as a result.

A determined adversary approaching a content-filtered system does not submit a single harmful request and give up if it is blocked. Instead, they probe the system systematically. They might start with a direct harmful request, observe that it is blocked, and then begin varying the phrasing. They shift to synonyms, then to euphemisms, then to hypothetical framings ("Imagine a character who explains..."), then to encoded representations, then to multi-turn strategies where each individual message appears benign but the conversation as a whole steers toward the harmful target.

This probing pattern means that a filter's worst-case performance is defined by the most persistent adversary, not the average user. AUC and F1 metrics computed on a fixed evaluation set tell you about the filter's performance against random or naive attempts. They say little about how long the filter holds up under systematic adversarial probing.

Quantifying adversarial robustness requires dedicated adversarial evaluation. The most useful metric is the "bypass rate at effort level $k$," which measures what fraction of harmful goals can be achieved by an adversary who makes at most $k$ attempts. This framing acknowledges that no filter is impenetrable but asks whether the filter raises the cost of successful evasion to a level that is operationally acceptable.

Understanding the most common evasion techniques helps you design filters that are harder to defeat. The main families are:

Lexical evasion targets keyword and pattern-based components. Character substitution (replacing "a" with "@", "e" with "3") defeats exact string matching. Adding spaces, hyphens, or zero-width characters between letters defeats simple regex. Deliberate misspellings that are phonetically similar to blocked terms defeat both. Unicode lookalike characters (Cyrillic letters that look like Latin letters to human readers but are different bytes to a naive string matcher) defeat character-level pattern matching.

Semantic evasion targets classifiers that have learned specific semantic patterns. Paraphrasing a harmful request into a vocabulary that the classifier has less experience with (e.g., clinical language for self-harm content, technical jargon for weapons information) can significantly reduce classifier confidence. Translation through a low-resource language and back ("laundering" the semantics) sometimes degrades classifier performance. Using euphemisms, coded language, or community-specific terminology that the training data did not cover is another form of semantic evasion.

Structural evasion targets models that classify short messages but struggle with long or unusual structures. Embedding a harmful request inside a much longer benign document (so the classifier averages over benign and harmful content) is one approach. Breaking a single harmful request into multiple turns, each of which appears benign in isolation, is another. Roleplaying framing ("You are a character who does not have safety restrictions") attempts to shift the model's generation mode rather than the filter's detection mode.

Meta-level evasion involves attacking the filter itself rather than trying to sneak through it. If an adversary can determine the classifier's category boundaries through probing, they can deliberately craft inputs that score just below the block threshold in all categories simultaneously. This "minimum-cost evasion" problem is related to adversarial examples in ML more broadly, and gradient-based attack methods can sometimes be adapted to text classification if the adversary has white-box access to the classifier.

Several techniques improve filter robustness against adversarial evasion:

Ensemble diversity makes it harder to simultaneously evade all components of a filter stack. If you combine a fast keyword filter, a TF-IDF classifier, a fine-tuned transformer, and an LLM judge, each with different architectures and training data, a bypass technique that defeats one component is less likely to defeat all of them simultaneously. The computational cost of maintaining a diverse ensemble is real, but so is the security benefit.

Adversarial training incorporates known bypass attempts as training examples for the classifier. After each red team exercise or real-world evasion incident, successful bypass examples are labeled and added to the training data for the next version of the filter. Over time, this process produces a classifier that has internalized a wide range of bypass techniques, though it will never be exhaustive.

Semantic normalization as a preprocessing step converts text to a canonical form before classification. Unicode normalization (converting lookalike characters to their standard forms), spell correction, and character-level deobfuscation all help ensure that surface-level manipulations do not defeat the underlying classifier. These normalization steps must be carefully designed to avoid introducing their own false positives (aggressive spell correction can mangle legitimate specialized vocabulary).

Rate limiting and behavioral signals complement text-based filtering by penalizing the probing behavior itself. An adversary who sends dozens of slightly varied harmful requests is exhibiting a behavioral pattern that is detectable regardless of whether any individual request is blocked. Combining text-based filtering with behavioral signals (request frequency, similarity of recent requests, account age and history) produces a more robust system than text filtering alone.

To make the interplay between threshold, class prevalence, and downstream impact concrete, let's work through a numeric example.

Suppose you operate a consumer chat service that handles 1 million requests per day. Based on manual review and historical data, you estimate that approximately 0.5% of requests are harmful (5,000 harmful requests per day). Your classifier has an AUC of 0.92, and you are considering two threshold settings.

Setting A: threshold = 0.3 (high sensitivity)

At this threshold, the classifier achieves recall $TPR = 0.92$ and false positive rate $FPR = 0.08$.

Expected blocks per day:

Precision at this operating point:

You are blocking 84,200 requests per day, but only 5.5% of those are harmful. Nearly 80,000 legitimate users per day have their requests incorrectly blocked. This is a severe user experience problem even though the filter is technically "catching 92% of harmful content."

Setting B: threshold = 0.7 (high precision)

At this threshold, recall drops to $TPR = 0.60$ and FPR drops to $FPR = 0.01$.

Expected blocks per day:

Precision at this operating point:

Now 23% of blocks are harmful content. You are blocking roughly 10,000 legitimate users per day, and missing 2,000 harmful requests. Whether this is acceptable depends on the harm categories involved: missing 2,000 hate speech messages per day might be tolerable if human review catches a fraction of them; missing 2,000 CSAM requests per day would not be.

This worked example illustrates why production filter configuration requires explicit harm modeling. The "best" threshold cannot be determined from AUC alone; it requires quantifying the number of false positives and false negatives at each operating point and weighing them against the operational costs of each error type.

Content filters are a necessary but imperfect tool, and deploying them without understanding their limitations leads to systems that are either too permissive or too restrictive.

The adversarial cat-and-mouse problem is perhaps the most important limitation. Every filter creates an implicit specification of what is blocked, and adversaries can probe the filter to learn its boundaries and find workarounds. This is especially true of keyword and pattern-based filters, but it applies to ML classifiers too: adversarial inputs crafted to evade a classifier are a well-documented phenomenon. A filter effective three months ago may be routinely bypassed today by adversaries who have reverse-engineered its decision boundaries. This is not a reason to abandon filtering, but it is a reason to treat filters as one layer in a defense-in-depth system rather than the sole safeguard.

Context blindness affects simple classifiers that operate on single messages. The same text can be harmful or benign depending on who sent it, what conversation preceded it, and what application it is used in. A classifier that ignores context will produce too many false positives in legitimate edge cases (medical information, fiction, security research, academic discussion) and miss harmful content that is benign on its surface but dangerous in context. Contextual classifiers that consider full conversation history reduce this problem but are more expensive to operate.

Minority language coverage is a systematic gap in most content filter training data. English-language harmful content is overrepresented in labeled datasets, and filters trained primarily on English perform worse on code-switched content, non-Latin scripts, and dialects. A filter that works well for English may effectively provide no protection for languages with limited labeled training data, creating inequitable protection across user populations.

False positive harm is real and sometimes overlooked. The cost of blocking a benign request is not zero. In healthcare applications, blocking a query about self-harm might prevent someone in crisis from reaching useful resources. In legal or academic contexts, blocking discussion of taboo topics prevents legitimate scholarship. Filters calibrated for one use case (general consumer chat) applied to a different use case (clinical mental health support) can cause direct harm through over-blocking. This argues for context-specific filter tuning rather than universal thresholds.

Computational costs at scale create real constraints on filter sophistication. At 1 million requests per day, adding 10 milliseconds of latency per request for an additional classifier adds up to nearly three CPU-hours of compute per day at minimum. LLM-based filters that add 500 milliseconds per request become prohibitively expensive at scale. These cost constraints push production systems toward fast, smaller classifiers for the common path, which inevitably sacrifice some accuracy. The cost-accuracy-latency tradeoff is fundamental, and there is no free lunch: you can have two of the three, but not all three simultaneously.

Definitional instability affects harm taxonomies over time. What counts as harmful speech evolves with social norms and legal changes. Categories that seemed clear when the filter was designed become contested, or new categories emerge that were not anticipated. A filter architecture that treats the taxonomy as fixed will gradually become misaligned with current definitions of harm as society changes around it. This argues for building filters on top of taxonomies that can be updated incrementally rather than architectures that bake the taxonomy into the model weights in ways that are difficult to revise.

Content filtering works best as one component in a broader safety stack, not as a standalone solution. The surrounding layers provide context that makes filtering more effective, and fallback when filtering is insufficient:

• Alignment training (RLHF, constitutional AI) reduces the base rate of harmful outputs, lowering the false negative rate that filtering must compensate for
• Input filtering catches obvious harmful requests before compute is wasted on model inference
• Output filtering catches failures that alignment training did not prevent
• Guardrails (next chapter) provide more sophisticated policy enforcement for complex multi-turn scenarios
• Human review provides ground truth for filter evaluation and catches the long tail of edge cases that automated systems miss
• Rate limiting and account controls reduce the impact of adversaries who probe the filter repeatedly

Understanding where content filtering fits in this stack helps you avoid the common mistake of over-relying on it or under-investing in complementary safeguards.

The deepest limitation of any purely automated filter is that it cannot make ethical judgments. It can only approximate the judgments that humans make when they review content. The quality of that approximation depends entirely on the quality and diversity of the humans who labeled the training data, the care with which the taxonomy was designed, and the rigor with which the filter's performance is evaluated and updated. A content filter codifies of human judgment rendered automatic, and it inherits both the wisdom and the biases of the humans who built it.

Content filtering is the automated detection and blocking layer that protects language model deployments from harmful inputs and prevents harmful outputs from reaching users. The two main technical approaches are classification-based filtering, which uses trained ML models to predict harm categories from text, and rule-based filtering, which uses keyword lists, regular expressions, and structured rules for patterns where precision and auditability matter most.

Filters can be placed before inputs reach the model (input filtering), after the model generates output (output filtering), or asynchronously for monitoring and retrospective analysis. Production systems typically use both input and output filters for defense in depth, accepting the higher total cost in exchange for broader coverage.

Filter evaluation requires more than standard classification metrics. Because harmful content is rare, evaluation sets must deliberately oversample harmful examples. Because adversaries adapt, red team evaluation is essential to measure filter effectiveness against motivated attackers. Because threats evolve, ongoing monitoring for distribution shift is necessary to keep filters current. The Bayesian relationship between classifier AUC, class prevalence, and precision means that filters deployed against rare harmful content will have much lower precision in production than holdout evaluation suggests, and this gap must be explicitly managed through threshold calibration.

The key limitations of content filtering are its adversarial vulnerability, context blindness in simple classifiers, uneven coverage across languages, the real cost of false positives, and the computational constraints that force tradeoffs between latency, cost, and accuracy at scale. These limitations argue for treating content filters as one layer in a defense-in-depth safety architecture rather than the sole safeguard against misuse.

The adversarial dynamics of content filtering, where adversaries probe and adapt their bypass techniques over time, mean that filter maintenance is an ongoing process rather than a one-time engineering task. Classifiers must be retrained regularly on new adversarial data. Taxonomies must evolve as definitions of harm change. Monitoring must surface failures quickly enough to respond before adversaries fully exploit them. The organizations that maintain effective content filtering over multi-year timelines are those that treat filtering as a living system requiring continuous investment, not a deployed artifact that can be set aside once it clears launch criteria.

In the next chapter, you will see how guardrails extend this architecture with more sophisticated multi-turn safety logic and policy enforcement, addressing some of the context-blindness limitations that simple content filters struggle with.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about content filtering.