Content Filtering › Implementation: A Classification-Based Content Filter
Michael BrenndoerferFebruary 22, 20266 min read
Language AI HandbookAll chapters Part 60 of 67Safety and Security
Content Filtering › Implementation: A Classification-Based Content FilterOne section · Sections in this chapter6 / 14 ↓
Sections in this chapter14
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's build a classification-based content filter from scratch to see how these concepts work in practice. We will use scikit-learn to train a logistic regression classifier on TF-IDF features from a synthetic toxicity dataset. This keeps the demonstration self-contained and executable without requiring a GPU or HuggingFace model download.
In[3]:
Codeimport warnings
import numpy as np
warnings.filterwarnings("ignore")
from sklearn.model_selection import train_test_split
# Synthetic training data: labeled examples for two categories
# toxic=1, safe=0
toxic_examples = [
"I hate you and everything you stand for",
"You are worthless garbage and should disappear",
"Die in a fire, nobody wants you here",
"You disgusting piece of trash",
"I will destroy you and make you suffer",
"You are the worst person alive, get lost",
"Go kill yourself, no one cares about you",
"You pathetic loser, your life is meaningless",
"Everyone hates you and you deserve it",
"I hope something terrible happens to you",
"You are a complete waste of oxygen",
"Shut up you stupid worthless idiot",
"I despise you with every fiber of my being",
"You make me sick, disappear forever",
"You are nothing and nobody likes you",
]
safe_examples = [
"How do I install Python on my computer?",
"What is the best way to learn calculus?",
"Tell me about the history of World War II",
"I need help with my homework assignment",
"What are some good recipes for dinner?",
"How does photosynthesis work?",
"Can you explain quantum mechanics simply?",
"What movies are popular right now?",
"I want to learn how to play guitar",
"How do I write a good cover letter?",
"What are the best hiking trails nearby?",
"Tell me about climate change and its effects",
"How do I improve my writing skills?",
"What programming languages should I learn?",
"Explain the difference between TCP and UDP",
"What is machine learning and how does it work?",
"How do neural networks learn from data?",
"I enjoy reading science fiction novels",
"What are some strategies for better sleep?",
"How do I start a vegetable garden?",
]
texts = toxic_examples + safe_examples
labels = [1] * len(toxic_examples) + [0] * len(safe_examples)
X_train, X_test, y_train, y_test = train_test_split(
texts, labels, test_size=0.3, random_state=42, stratify=labels
)In[4]:
Codefrom sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import Pipeline
# Build a TF-IDF + Logistic Regression pipeline
# TfidfVectorizer captures word importance; LogisticRegression gives calibrated probabilities
clf_pipeline = Pipeline(
[
(
"tfidf",
TfidfVectorizer(ngram_range=(1, 2), min_df=1, max_features=5000),
),
("clf", LogisticRegression(C=1.0, max_iter=1000, random_state=42)),
]
)
clf_pipeline.fit(X_train, y_train)
y_pred = clf_pipeline.predict(X_test)
y_prob = clf_pipeline.predict_proba(X_test)[:, 1]
auc_score = roc_auc_score(y_test, y_prob)Out[5]:
ConsoleAUC-ROC: 0.850
precision recall f1-score support
safe 0.83 0.83 0.83 6
toxic 0.80 0.80 0.80 5
accuracy 0.82 11
macro avg 0.82 0.82 0.82 11
weighted avg 0.82 0.82 0.82 11The classifier achieves high AUC on this synthetic dataset. In practice, training data diversity is critical: a model trained only on the most obvious toxic examples will fail on subtler or paraphrased variations. Now let's build the threshold-based filter on top.
In[6]:
Codedef content_filter(texts_to_check, pipeline, threshold=0.5):
"""
Apply a threshold to classifier probabilities to make block/pass decisions.
Returns a list of dicts with text, score, and block decision.
"""
probs = pipeline.predict_proba(texts_to_check)[:, 1]
results = []
for text, prob in zip(texts_to_check, probs):
results.append(
{
"text": text,
"toxicity_score": float(prob),
"blocked": prob >= threshold,
}
)
return results
# Test the filter on held-out examples
test_inputs = [
"How do I install Python on my computer?",
"You are worthless and nobody likes you",
"Tell me about the history of World War II",
"I despise you and hope you suffer",
"What programming languages should I learn?",
]
filter_results = content_filter(test_inputs, clf_pipeline, threshold=0.5)Out[7]:
ConsoleText (truncated) Score Decision ----------------------------------------------------------------- How do I install Python on my computer? 0.305 allowed You are worthless and nobody likes you 0.590 BLOCKED Tell me about the history of World War II 0.338 allowed I despise you and hope you suffer 0.557 BLOCKED What programming languages should I learn? 0.315 allowed
The filter correctly blocks toxic content and passes safe content through. Now let's explore how the threshold choice affects the precision-recall tradeoff by computing metrics across a range of thresholds.
In[8]:
Code# Compute metrics at different thresholds on the test set
thresholds_to_test = [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]
threshold_metrics = []
y_prob_test = clf_pipeline.predict_proba(X_test)[:, 1]
y_test_arr = np.array(y_test)
for t in thresholds_to_test:
preds = (y_prob_test >= t).astype(int)
tp = int(np.sum((preds == 1) & (y_test_arr == 1)))
fp = int(np.sum((preds == 1) & (y_test_arr == 0)))
fn = int(np.sum((preds == 0) & (y_test_arr == 1)))
tn = int(np.sum((preds == 0) & (y_test_arr == 0)))
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
f1 = (
2 * precision * recall / (precision + recall)
if (precision + recall) > 0
else 0.0
)
fpr = fp / (fp + tn) if (fp + tn) > 0 else 0.0
threshold_metrics.append(
{
"threshold": t,
"precision": precision,
"recall": recall,
"f1": f1,
"false_positive_rate": fpr,
}
)Out[9]:
ConsoleThreshold Precision Recall F1 FP Rate -------------------------------------------------------- 0.2 0.455 1.000 0.625 1.000 0.3 0.500 1.000 0.667 0.833 0.4 0.625 1.000 0.769 0.500 0.5 0.800 0.800 0.800 0.167 0.6 0.500 0.200 0.286 0.167 0.7 0.000 0.000 0.000 0.000 0.8 0.000 0.000 0.000 0.000
Notice the clear tradeoff: lowering the threshold increases recall (we catch more harmful content) but also raises the false positive rate (more benign content is blocked). Threshold 0.3 catches nearly all harmful content at the cost of blocking some safe requests; threshold 0.7 produces almost no false positives but misses a significant fraction of harmful inputs. The optimal threshold depends on which type of error is more costly for your application.
Key Parameters
The key parameters for a classification-based content filter are:
- model architecture: TF-IDF + logistic regression for maximum speed and interpretability; fine-tuned transformer encoders (BERT, RoBERTa) for better accuracy on context-dependent content; LLMs for contextual multi-turn reasoning.
- harm taxonomy: The set of categories the classifier detects. More categories require more labeled training data per category.
- threshold: The probability cutoff controlling precision vs. recall. Should be tuned per category based on the business cost of each error type.
- ngram_range (for TF-IDF): Including bigrams and trigrams captures multi-word phrases that single-word features miss.
- context window: Whether the classifier sees only the current message or the full conversation history. Wider context reduces false positives and negatives on context-dependent content.