Content Filtering › Worked Example: Estimating Filter Impact

To make the interplay between threshold, class prevalence, and downstream impact concrete, let's work through a numeric example.

Suppose you operate a consumer chat service that handles 1 million requests per day. Based on manual review and historical data, you estimate that approximately 0.5% of requests are harmful (5,000 harmful requests per day). Your classifier has an AUC of 0.92, and you are considering two threshold settings.

Setting A: threshold = 0.3 (high sensitivity)

At this threshold, the classifier achieves recall $TPR = 0.92$ and false positive rate $FPR = 0.08$.

Expected blocks per day:

$$\begin{aligned} \text{True positives} &= 5{,}000 \times 0.92 = 4{,}600 \\ \text{False positives} &= 995{,}000 \times 0.08 = 79{,}600 \\ \text{Total blocks} &= 84{,}200 \end{aligned}$$

Precision at this operating point:

$$\text{Precision} = \frac{4{,}600}{4{,}600 + 79{,}600} \approx 0.055$$

You are blocking 84,200 requests per day, but only 5.5% of those are harmful. Nearly 80,000 legitimate users per day have their requests incorrectly blocked. This is a severe user experience problem even though the filter is technically "catching 92% of harmful content."

Setting B: threshold = 0.7 (high precision)

At this threshold, recall drops to $TPR = 0.60$ and FPR drops to $FPR = 0.01$.

Expected blocks per day:

$$\begin{aligned} \text{True positives} &= 5{,}000 \times 0.60 = 3{,}000 \\ \text{False positives} &= 995{,}000 \times 0.01 = 9{,}950 \\ \text{Total blocks} &= 12{,}950 \end{aligned}$$

Precision at this operating point:

$$\text{Precision} = \frac{3{,}000}{3{,}000 + 9{,}950} \approx 0.232$$

Now 23% of blocks are harmful content. You are blocking roughly 10,000 legitimate users per day, and missing 2,000 harmful requests. Whether this is acceptable depends on the harm categories involved: missing 2,000 hate speech messages per day might be tolerable if human review catches a fraction of them; missing 2,000 CSAM requests per day would not be.

This worked example illustrates why production filter configuration requires explicit harm modeling. The "best" threshold cannot be determined from AUC alone; it requires quantifying the number of false positives and false negatives at each operating point and weighing them against the operational costs of each error type.

---