Content Filtering › Classification-Based Filtering

The most widely deployed approach to content filtering is to treat it as a supervised classification problem. You define a set of harm categories, collect labeled examples, and train a model that predicts which (if any) categories apply to a given text.

Before you can train a classifier, you need to decide what you are classifying. This requires a harm taxonomy: a structured list of categories that represent the kinds of content you want to block.

Different organizations have settled on different taxonomies, but common categories include:

• Hate speech: Content that degrades or dehumanizes individuals or groups based on protected characteristics such as race, religion, gender, or sexual orientation.
• Harassment: Targeted threats, insults, or intimidation directed at specific people.
• Sexual content: Explicit sexual material, which may have subcategories for adult-only content versus content involving minors (which is universally prohibited).
• Violence: Graphic descriptions of physical harm, instructions for violent acts, or glorification of violence.
• Self-harm: Content that promotes, instructs, or glorifies suicide or self-injury.
• Dangerous information: Instructions for weapons, drugs, malware, or other content that enables real-world harm.
• Spam and manipulation: Scam content, phishing attempts, or coordinated inauthentic behavior.

The boundary between categories is often fuzzy. A piece of text might be simultaneously violent and hateful. A description of drug effects might be educational in one context and dangerous in another. For this reason, most production classifiers are trained as multi-label classifiers rather than single-label ones: a single input can trigger multiple categories simultaneously.

OpenAI's moderation API, for example, uses categories like hate, hate/threatening, harassment, harassment/threatening, self-harm, self-harm/intent, self-harm/instructions, sexual, sexual/minors, violence, and violence/graphic. The hierarchical structure (e.g., self-harm/intent is more specific than self-harm) allows downstream systems to apply different thresholds to different severity levels.

Designing a good harm taxonomy is harder than it looks. Categories must be specific enough to be trainable (you need examples that clearly fall in or out of each category) but broad enough to cover the range of harmful content in each semantic family. If a category is too narrow, adversaries can easily rephrase their way around it. If it is too broad, the classifier will block too much legitimate content. The process of building a taxonomy often reveals implicit assumptions about what counts as harmful, and surfacing those assumptions explicitly is one of the most valuable outcomes of the taxonomy-building exercise.

Taxonomies also need to be revisited regularly. Harm categories that were well-defined two years ago may need subdivision as new attack patterns emerge, or may need merger as the operational cost of maintaining separate classifiers becomes prohibitive. Organizations that treat their taxonomy as a fixed artifact rather than a living document tend to find that their filters gradually drift out of alignment with actual harm patterns in production.

A classification filter is typically a fine-tuned language model, though simpler text classifiers are used when latency is critical. The training pipeline involves three main components.

Data collection is the hardest part. You need examples of both harmful and benign content that reflect the actual distribution you will encounter in production. Harmful examples come from multiple sources: red team exercises (where human testers deliberately generate boundary cases), human-written datasets like the HateXplain corpus or ToxiGen, synthetic generation (where you prompt a capable model to produce borderline examples and then filter them), and production logs from real deployments (with appropriate privacy protections). The data must be carefully balanced, because harmful content is rare by definition. A model trained on a corpus where only 0.1% of examples are harmful will learn to predict "safe" for everything and achieve 99.9% accuracy while being completely useless.

The diversity of training data deserves special emphasis. A dataset that captures only the most obvious, explicit forms of harmful content will produce a classifier that is easily fooled by subtle paraphrases, euphemisms, or culturally specific expressions. Red team annotators with adversarial mindsets, subject matter experts in specific harm domains (e.g., child safety experts for CSAM-related categories, counter-terrorism analysts for violent extremism), and crowdsourced annotation from diverse populations all contribute data characteristics that no single source can provide on its own.

Labeling is the process of assigning harm categories to examples. Because the category boundaries are fuzzy and sometimes contested, labeling requires clear guidelines, multiple annotators per example, and inter-annotator agreement metrics to catch cases where annotators disagree. Disagreement is often informative: a high inter-annotator disagreement rate on a category suggests that the category is ambiguously defined and needs refinement.

Inter-annotator agreement is commonly measured using Cohen's kappa coefficient $\kappa$, which corrects for the probability of chance agreement:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

where:

• $p_o$: the observed proportion of cases where annotators agree
• $p_e$: the expected proportion of cases where annotators agree by chance, computed from the marginal distributions of each annotator's labels

A $\kappa$ value near 1.0 indicates near-perfect agreement; a value near 0.0 indicates agreement no better than random chance; negative values indicate systematic disagreement. For safety classification tasks, a $\kappa$ below 0.6 on a given category is typically a signal to revisit the labeling guidelines for that category before using the data for training.

Model architecture for filtering classifiers typically follows one of three patterns. A bidirectional encoder like BERT or RoBERTa is fine-tuned on the classification task. The encoder reads the full text, produces a [CLS] token embedding, and a classification head on top predicts the harm category probabilities. Alternatively, a full autoregressive model (GPT-class) can be prompted or fine-tuned to output category labels. Finally, for embedding-based approaches, you can use a pretrained text encoder to project texts into a vector space and then apply a lightweight classifier (logistic regression, SVM) on top of the embeddings.

The advantage of encoder-based classifiers is speed and simplicity. A fine-tuned BERT model running on GPU can classify tens of thousands of texts per second. The advantage of LLM-based classifiers is nuance: large models better understand context and sarcasm across a multi-turn conversation. Many systems use both: a fast classifier for the common case and a more expensive LLM-based check for high-stakes or uncertain cases.

A third option that is gaining traction is using a large LLM as a zero-shot or few-shot judge, where you prompt the model with the harm taxonomy and a few labeled examples and ask it to evaluate each new input. This approach requires no fine-tuning and can be updated by modifying the prompt rather than retraining, which makes it very agile for responding to new threat patterns. The cost is high: LLM inference is orders of magnitude more expensive than a small fine-tuned encoder, and running an LLM judge on every request is only practical at low request volumes or when the cost of false negatives is extremely high. In practice, the LLM-as-judge pattern is often deployed asynchronously for quality monitoring rather than in the real-time request path.

A classification model outputs probabilities, not hard labels. You must choose a threshold above which a probability triggers a block. This choice is a policy decision, not a technical one.

Setting the threshold involves a fundamental tradeoff between two types of errors:

• False positives (blocking benign content): These frustrate users, degrade the product experience, and, in some cases, cause real harm. A medical information service that blocks all mentions of "overdose" will fail users seeking harm-reduction information. A creative writing platform that blocks all violent content will frustrate novelists. False positives erode user trust.
• False negatives (allowing harmful content through): These represent the filter's failure to catch what it was designed to catch. The consequences depend on the category and use case: allowing one piece of hate speech in a consumer product is bad; allowing a bioweapon synthesis request in a research API is catastrophic.

The right threshold depends on your use case, your user population, and your harm model. A children's education platform should set a very low threshold for sexual content (high sensitivity) and accept a higher false positive rate. An adult content platform might set a very high threshold for most harm categories and only strictly enforce legal violations. Most platforms adjust thresholds per category rather than applying a single global threshold.

It is useful to frame threshold selection as an optimization problem. Define the business cost $C_{FP}$ of a false positive and the business cost $C_{FN}$ of a false negative for each harm category. The optimal threshold $\tau^*$ minimizes expected total cost:

$$\tau^* = \underset{\tau}{\operatorname{argmin}} \; \mathbb{E}\left[C_{FP} \cdot \mathbf{1}[\hat{y}(\tau) = 1, y = 0] + C_{FN} \cdot \mathbf{1}[\hat{y}(\tau) = 0, y = 1]\right]$$

where $\hat{y}(\tau) = \mathbf{1}[p(y=1|x) \geq \tau]$ is the predicted label at threshold $\tau$. In practice, $C_{FP}$ and $C_{FN}$ are rarely quantified precisely, but the framing is useful: it forces you to articulate the relative cost of each error type and makes the policy nature of threshold selection explicit. A threshold of 0.5 is not a neutral default; it implicitly assumes the cost of a false positive equals the cost of a false negative, which is almost never true in safety applications.

Text that is harmful in one context may be benign in another. The phrase "how do I kill this process?" is dangerous in a conversation about biological weapons but completely normal in a software debugging context. Contextual classifiers address this by including conversation history, system prompt, and metadata (user role, application type, API tier) as input to the classification model rather than relying on the single message.

This is one of the key advantages of LLM-based classifiers over simple text classifiers. When you feed a whole conversation thread to a capable language model with the prompt "Is this exchange harmful?", the model can use the full context to make a more context-sensitive judgment. Simple bag-of-words or keyword-based classifiers cannot.

The downside is latency and cost. Running a full LLM classifier on every message of a high-volume API adds significant overhead. In practice, many systems use a tiered approach: a fast low-latency classifier runs on every request, and a slow high-quality classifier runs only on requests that the fast classifier flagged as uncertain (probability between 0.3 and 0.7, for example) or on a random sample for quality monitoring.

Context windows also create privacy risks. If you feed the full conversation history to a classifier API, you are transmitting potentially sensitive user content to the classification service. In some jurisdictions, this raises regulatory compliance questions. Architectures that keep classification in-house (running the classifier on your own infrastructure) avoid this concern but require more operational overhead than calling a third-party API.

Metadata signals can help contextual classification even when full conversation history is not available. Knowing that a request comes from an API tier that requires identity verification, from an account that has been active for three years, or from a system prompt that authorizes adult content significantly shifts the prior probability of harm. Some filter architectures encode these signals as additional input features to the classifier, rather than requiring the full context to be re-analyzed linguistically.

It is worth being precise about what "multi-label" means here, because it differs in important ways from ordinary multi-class classification. In a standard multi-class classifier, each input belongs to exactly one class: a digit image is a 3 or a 7, not both. In multi-label classification, each input can belong to any subset of the available categories simultaneously. A message can be both hateful and threatening; a piece of text can contain both sexual content and violence.

This distinction has practical implications for how you build and calibrate the classifier. In a multi-class setting, you apply a softmax activation over the output logits to produce a probability distribution that sums to 1.0 across all classes. In a multi-label setting, you apply a sigmoid activation to each logit independently, producing probabilities that are each in the range $(0, 1)$ and do not sum to any fixed total. Each category has its own independent probability estimate. You then apply a per-category threshold to decide which labels to assign.

The sigmoid function applied to logit $z_k$ for category $k$ is:

$$p_k = \sigma(z_k) = \frac{1}{1 + e^{-z_k}}$$

Each $p_k$ is treated as an independent binary probability. The label prediction for category $k$ at threshold $\tau_k$ is then simply:

$$\hat{y}_k = \mathbf{1}[p_k \geq \tau_k]$$

Because the thresholds $\tau_k$ are per-category, you tune them independently based on the operational requirements for each harm type. The threshold for violence/graphic in a general consumer app might be 0.3, while the threshold for hate/non-threatening in a research API might be 0.8.

The independence assumption in multi-label classification is a simplification. Categories are not truly independent: a text that scores high for "threat" is more likely to also score high for "harassment" than a random text. Some systems model these correlations explicitly using label-correlation architectures or by chaining classifiers (first predict the primary category, then predict secondary categories conditioned on the primary). In practice, the simple independent sigmoid approach works well enough for most use cases, because the categories are correlated in ways the model learns during training regardless.

---