Classification Metrics and Decision Thresholds › Example

The simulated data shows the machinery at scale, but its numbers arrive from code. A tiny hand-worked case shows where each number comes from and makes the formulas concrete. Sixteen accounts have been scored, four of them are fraudulent, and the scores are sorted from highest to lowest. Every number below is small enough for you to check with a calculator.

There are $N_1 = 4$ positives and $N_0 = 12$ negatives, so the prevalence is $\pi = 4/16 = 0.25$. The costs are $c_{FP} = 5$ dollars for reviewing an account that turns out to be clean and $c_{FN} = 150$ dollars for missing a fraudulent one. In other words, each false positive costs 5 dollars and each false negative costs 150 dollars.

Start at the default threshold $t = 0.50$. The rule flags the first six accounts, since their scores are at least 0.50. Of those six, three are fraudulent (ranks 1, 2, 4) and three are not (ranks 3, 5, 6). The one remaining fraud sits at rank 7 with a score of 0.45, so it is missed. These four counts are exactly the four cells of the confusion matrix, and they are complete: every account appears once.

$$\mathbf{C}(0.50) = \begin{bmatrix} TN & FP \\ FN & TP \end{bmatrix} = \begin{bmatrix} 9 & 3 \\ 1 & 3 \end{bmatrix}.$$

The metrics follow mechanically. Precision is $3/(3+3) = 0.500$, recall is $3/(3+1) = 0.750$, the false positive rate is $3/(3+9) = 0.250$, and $F_1 = 2 \times 0.5 \times 0.75/(0.5 + 0.75) = 0.600$, which the count form confirms: $2 \times 3/(2 \times 3 + 3 + 1) = 6/10$. The expected cost per account is

$$\hat{C}(0.50) = \frac{5 \times 3 + 150 \times 1}{16} = \frac{165}{16} = 10.3125.$$

Now lower the threshold to 0.45, which adds rank 7 to the alert queue. Nothing else changes, because no other score lies between 0.45 and 0.50. The confusion matrix becomes

$$\mathbf{C}(0.45) = \begin{bmatrix} 9 & 3 \\ 0 & 4 \end{bmatrix},$$

so precision rises to $4/7 \approx 0.571$, recall rises to $4/4 = 1.000$, the false positive rate is unchanged at $3/12 = 0.250$, and $F_1 = 2 \times 4/(2 \times 4 + 3 + 0) = 8/11 \approx 0.727$.

where:

• $TP = 4$: the number of flagged frauds after including rank 7
• $FP = 3$: the number of flagged clean accounts (ranks 3, 5, 6)
• $FN = 0$: no fraud remains unflagged
• $TN = 9$: the number of unflagged clean accounts

The cost drops to

$$\hat{C}(0.45) = \frac{5 \times 3 + 150 \times 0}{16} = \frac{15}{16} = 0.9375.$$

Recall rises and precision also rises here because the one added alert is a true positive, which is the opposite of the general tendency: we noted earlier that raising the threshold from 0.45 to 0.50 reduced precision from 4/7 to 1/2 while reducing recall. When moving toward lower thresholds, adding a true positive can lift both metrics, while adding a false positive leaves recall unchanged and lowers precision when the queue already contains a true positive. This example shows why precision need not move in the opposite direction from recall. The cost, however, falls by more than a factor of ten. Multiplying this observed difference by a hypothetical 10,000 accounts gives $93,750. Interpreting that as a future daily saving would additionally require the example error rates and stated costs to transfer to the future stream.

The threshold need not be exactly 0.45. Any value in the interval between the seventh and eighth scores, $t \in (0.38, 0.45]$, produces the same predictions and therefore the same cost. This is the cost plateau of the previous section, and it means the reported threshold should be understood as a representative of an interval. The midpoint rule used in the code would report 0.415.

Next we check the curve summaries against the same sixteen points. Ordering the positives by rank, the true positive count after each is $1, 2, 3, 4$ at ranks $1, 2, 4, 7$, so recall takes the values 0.25, 0.50, 0.75, and 1.00 with precisions 1/1, 2/2, 3/4, and 4/7. Average precision sums the recall increments weighted by the precision achieved:

$$\begin{aligned} \text{AP} &= \tfrac14\!\left(1+1+\tfrac34+\tfrac47\right) \\ &= \tfrac12+\tfrac{3}{16}+\tfrac17 \\ &= \tfrac{93}{112} \approx 0.8304. \end{aligned}$$

For the AUC we count, for each positive, how many negatives it outranks. Rank 1 outranks all twelve, rank 2 also outranks all twelve, rank 4 outranks the eleven negatives below it, and rank 7 outranks the nine below it:

$$\widehat{\text{AUC}} = \frac{12 + 12 + 11 + 9}{12 \times 4} = \frac{44}{48} \approx 0.9167.$$

The trapezoidal area under the same precision-recall points is a different number. The horizontal segments contribute $0.25 \times 1$ and $0.25 \times 1$, the segment from recall 0.50 to 0.75 connects precision 2/3 to 3/4, and the segment from 0.75 to 1.00 connects precision 1/2 to 4/7:

$$\widehat{\text{PR}}_{\text{trap}} = 0.25 + 0.25 + 0.25\!\left(\tfrac{2/3 + 3/4}{2}\right) + 0.25\!\left(\tfrac{1/2 + 4/7}{2}\right) \approx 0.8110.$$

The two do not agree in this case; the trapezoid differs from average precision. Here average precision is about 0.8304 and the trapezoidal area about 0.8110, a difference of $13/672\approx0.0193$. The definitions can sometimes yield the same number, even when precision is not constant, because segment differences can cancel. Name the convention rather than treating the two definitions as interchangeable.

Finally, the closed form for calibrated scores gives $t^{*} = 5/(5+150) \approx 0.0323$. Applying it here would flag every account scoring at least 0.0323, which is ranks 1 through 15, giving $TP = 4$ and $FP = 11$ for a cost of $(5 \times 11 + 150 \times 0)/16 = 3.4375$. The toy scores are used here as an arbitrary ranking, so we have not established the conditional-probability assumption for that cutoff. Its observed cost is larger than the empirical optimum of 0.9375. That finite-sample comparison alone would not prove miscalibration: a sample-selected rule can also beat a population-optimal rule on the sample used to select it.

We close the example by checking the hand arithmetic against the code, which is the habit that keeps your worked examples honest.

The preceding code checks the hand calculation against the same definitions. Agreement on this example is a useful implementation check, rather than proof that every later use is correct.

The four counts at the default threshold can be read as one grid. This grid summarizes the aggregate unweighted binary outcomes at that cut, before any ratio such as precision or recall is formed.