Classification Metrics and Decision Thresholds › Visualizing Classification Metrics and Thresholds

We now build a small alerting problem where every quantity is known by construction. Each account has two features, the linear predictor is $z_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2}$, the label is drawn as $y_i \sim \text{Bernoulli}(p_i)$ with $p_i = 1/(1+e^{-z_i})$, and the model's score is exactly $p_i$, the probability you would threshold. Because the label was generated from the same probability the score reports, the score is calibrated by construction. That lets us compare a finite-sample empirical cost minimum with the population cutoff from the previous section. They need not agree exactly on a realized sample; an arbitrary score would not justify the analytic probability cutoff at all.

The scores arrive from a model that would, in a real pipeline, have been fitted on a separate training split. Since we start from scores, the roles you need here are selection (validation) and reporting (test). We look at the two samples side by side to confirm they are drawn from the same population.

The two splits are drawn from the same generating process and land close on size, prevalence, and mean score. Small differences reflect sampling variation, so the validation and test sets can reasonably be treated as exchangeable draws from one population for the purposes of this example.

Next we write the sweep that does the work. These tutorial functions assume paired one-dimensional nonempty arrays with binary labels, both classes present and finite scores in [0, 1], as used in this chapter. They do not implement a general input-validation API. Everything below is a direct transcription of the formulas: counts at a single threshold, counts for every distinct threshold at once using cumulative sums over the score-sorted data, the two curves, and the cost minimization.

We compute the curve arrays and operating metrics before the figures and reuse them in the plotting cells.

The first picture is the raw material: how the scores are distributed in each class. The vertical axis is logarithmic so that density values across a wide range remain visible. Each class histogram is normalized separately, so the height comparison describes class-conditional density rather than the ratio of class headcounts.

The two distributions overlap almost everywhere, which is the whole difficulty. The negative density is concentrated below 0.05, but its tail is long, so a threshold placed low enough to catch most positives drags in a large absolute number of negatives. The default cut of 0.50 sits far out in the positive tail. It would raise very few alerts and miss most fraud. The cost-optimal threshold sits far below it, inside the overlap, which is the quantitative version of "we would rather review more accounts than miss the fraud".

Plotting the metrics against the threshold itself makes the tradeoff explicit. The figure below is a useful diagnostic for a non-technical audience, because the horizontal axis is the one thing they control.

Read the left edge first. When the threshold approaches zero every account is flagged, recall is 1, the false positive rate is 1, and precision collapses to the prevalence, which is the dotted line: if you alert on everything, the fraction of alerts that are real is exactly the base rate. Read the right edge next. When the threshold exceeds the largest score nothing is flagged and precision is 1 by convention. The interesting region is the middle, and the dashed line marks where the cost function prefers to sit. In this realization, the selected threshold lies below the recall–precision crossing. The crossing is determined by the scores and labels; the error costs choose where the selected marker sits relative to it.

The ROC and precision-recall curves are the same sweep with the threshold eliminated. The first figure adds the iso-cost line from the population cost formula; its slope encodes the ratio of false positive cost to false negative cost weighted by prevalence. The second figure shows the same classifier in precision-recall space, where the horizontal line is the prevalence. In the population, a score independent of the label has precision equal to prevalence at thresholds with positive alert probability. A finite empirical curve can fluctuate around that baseline, and its no-alert corner still uses the stated plotting convention.

::: {layout-ncol=2}

:::

The ROC curve shows a healthy ranking: the area is well above the diagonal. The operating point, however, sits at a false positive rate that will look alarming to you next to the default square marker. That is the correct reading. With a miss costing thirty times a false alarm, the optimum tolerates many false alarms, and the iso-cost line makes the arithmetic visible: in the (FPR, TPR) plane its positive slope is $c_{FP}(1-\pi)/(c_{FN}\pi)$. The line passes through the minimum-cost empirical point and supports the available ROC points: a point above it would have smaller stated cost. A smooth tangency is not required for a discrete curve.

The precision-recall figure tells the same story in the units the review team cares about. Precision at the operating point is modest, but the alternative at the default threshold has recall of only 0.25 and therefore misses three quarters of the frauds, so its higher precision is largely irrelevant. Precision alone does not identify a useful rule, and it need not rise when the threshold rises. In the worked example below, increasing the threshold from 0.45 to 0.50 reduces precision from 4/7 to 1/2 as well as reducing recall from 1.000 to 0.750, because rank 7 (a true positive) drops out of the alert queue.

The final figure of this section plots the cost itself, and it is where the selection decision actually happens. The validation curve is what we minimize; the test curve evaluates fixed candidate rules on held-out observations and would stay unopened until after the validation choice was recorded. One realized test curve does not prove that the validation minimizer is the population optimum. The test curve's minimum and its value at the selected threshold are descriptive of this split, not an independent re-selection; the test set is used only to evaluate the fixed validation-selected rule.

The empirical cost curve is a step function because predictions change only when a score is crossed. A plateau between adjacent distinct scores has width equal to their numerical gap. The cost jump at a score depends on the labels and number of observations tied there, as well as the stated error costs; neither fixes the shape of the minimum on its own. Inspect both the cost differences and threshold values; curve shape alone does not establish stability across new samples.

The two curves nearly coincide, which is the optimistic but expected outcome here: validation and test are drawn from the same distribution and both are large. The closed-form threshold for calibrated scores lands close to, but not exactly on, the empirical minimizer, because the empirical minimizer is a noisy estimate. Finite-sample differences are possible despite calibration. Convergence to one numeric threshold additionally requires conditions such as a unique population minimizer and suitable sampling; calibration alone does not establish those conditions.

The invariance claims of the previous section deserve a picture. The following reweights the validation sample so that the positive share becomes 20 percent and then 2 percent, holding the class-conditional score distributions fixed. The weights are constant within each class, and because the ROC coordinates normalize within each class, the ROC curve cannot change; precision, which mixes the classes, changes at mixed operating points with both true and false positives. Pure-positive prefixes and the plotting endpoint can stay unchanged.

::: {layout-ncol=2}

:::

The two ROC curves lie on top of each other and the two AUC values agree to three decimals, while the average precision values differ substantially. Both plots are honest; they answer different questions. The ROC answer is "how well does this score order the two classes", which does not depend on how the classes are mixed. The precision-recall answer is "how useful would this score be here", which very much does. If we tuned the alerting model on a sample with 20 percent positives and deployed it where positives are 2 percent, the ROC curve would have told us nothing was wrong.