Classification Metrics and Decision Thresholds

A risk model produces a score; an alerting rule decides which scores lead to action; you choose where that cut sits. Suppose a transaction has a true conditional fraud probability of 0.04, a false alarm costs $5 and a missed fraud costs $150. With zero cost for correct decisions, alerting has expected cost 0.96 times $5, or $4.80, while letting it pass has expected cost 0.04 times $150, or $6. Under these assumptions, alerting costs less. An arbitrary score of 0.04 does not establish that probability: both the score scale and the consequences of the decision matter.

Class imbalance makes accuracy alone a poor guide to a decision. Suppose one transaction in fifty is fraudulent. A model that flags nothing is right 98 percent of the time but misses every fraud. A model that flags everything catches every fraud and sends every transaction for review. Accuracy happily reports 98 percent for the first and 2 percent for the second, so we need measurements that separate the two kinds of mistakes instead of merging them into one number.

We fix notation once and use it throughout. We start with the confusion matrix, the four counts every binary decision produces, and read precision, recall, and the false positive rate directly off it. We then follow those quantities as the threshold slides, which produces the threshold curves, the ROC curve, and the precision-recall curve, along with their summary numbers, ROC AUC and average precision. Finally we attach costs to the two mistakes and show that the threshold is not a matter of taste but an optimization problem: minimize observed cost on validation data, then evaluate the fixed rule once on test data.

Two distinctions run through the whole chapter and are worth stating early for you. The first is between ranking and calibration. ROC AUC and average precision care only about the order in which a model sorts examples; they are unchanged if we replace every score by its cube. The analytic probability cutoff needs a meaningful probability scale; empirical cost selection can instead evaluate arbitrary scores directly. The second is between measurement and decision. Metrics summarize how a fixed classifier behaves; a threshold turns a score into a classifier. Choosing the threshold is model selection in miniature, and the same leakage rules from Validation Strategies and Data Leakage apply: the data used to pick the threshold cannot also provide an independent evaluation of that choice.

Metrics built on the confusion matrix tell us things accuracy hides. Precision answers the question the review team actually asks, which is what fraction of the alerts you are about to work through are real. Recall answers the question you ask as a risk owner, which is what fraction of the bad events we are catching. Keeping these separate makes the tradeoff visible instead of averaging it away, and it makes the tradeoff negotiable: we can move along the curve deliberately rather than discovering the operating point by accident.

The threshold is an inexpensive and effective control you can turn. Once a model is trained, moving the threshold changes the operating point without retraining, and the whole family of achievable confusion matrices can be read off in a single sweep of order $n \log n$ operations. For a fixed score and the inclusive threshold convention, the sweep enumerates every achievable deterministic threshold state, including the endpoints and whole tied groups.

Curve summaries are useful for comparing models before you fix any threshold. ROC AUC has a probabilistic meaning: the probability that an independent positive outranks an independent negative, with half credit for ties. It is invariant to strictly increasing transformations of the score, so a model is not penalized for being badly calibrated when the question is only about ordering. Average precision focuses the same idea on the positive class and is more informative when positives are rare, because it is sensitive to the false positives that actually compete with true positives for review capacity.

Cost-based threshold selection converts an abstract tradeoff into a decision. Naming $c_{FP}$ and $c_{FN}$ forces the organization to state what each kind of mistake costs, and the population objective identifies a threshold that minimizes expected cost, while its empirical estimate selects a threshold from validation observations. When scores are calibrated, the optimal threshold has a closed form, $c_{FP}/(c_{FP}+c_{FN})$, which is easy to audit and easy to challenge in a meeting.

Every one of these metrics is a sample estimate, and at low prevalence the estimates are noisy precisely where it counts. Precision has predicted positives as its denominator, so few alerts can make its estimate sensitive to sampling. AUC uncertainty depends on both class counts and the score distributions. A point estimate describes the observed result but does not describe that uncertainty.

ROC AUC's indifference to prevalence is a feature for ranking comparisons and a trap for decision making. Because the false positive rate divides by the number of negatives, a model can achieve a flattering AUC while producing an unusable number of false alerts when negatives vastly outnumber positives. A high AUC also says nothing about the operating point we intend to use: two models with identical AUC can behave very differently in the low false positive rate region where a real review team lives.

Average precision fixes the prevalence problem in one direction and creates it in another. It depends on the class balance of the evaluation set, so an average precision of 0.6 measured on a sample with 5 percent positives cannot be assumed to hold at 1 percent prevalence. Re-evaluate or adjust under justified transport assumptions; a change is not inevitable, as perfect separation illustrates.

Calibration is a separate property that neither AUC nor average precision measures. Both are ranking statistics, blind to any strictly increasing rescaling of the scores. Empirical cost minimization can use a rank-only score, whereas the closed-form rule $c_{FP}/(c_{FP}+c_{FN})$ requires the stated probability assumption. A value between 0 and 1, including a vote fraction, is not by itself evidence of calibration for the deployment population.

Finally, threshold selection is itself a fitting procedure, so it consumes data. For a fixed candidate family whose individual validation cost estimates are unbiased, the expected validation minimum is no greater than the smallest population cost in that family. This expectation does not dictate the ordering in one realized validation/test comparison. The size of selection optimism depends on the joint variability of the estimates and the selection procedure; a flat cost curve alone does not bound it. Independent held-out observations evaluate the selected fixed rule.

A fitted classifier has not yet made a decision. It has assigned each example a score, and those scores induce an ordering. To act on that ordering we choose a cut point in the ranked list and alert on everything above it. That cut point is the threshold, and the formulas below are the vocabulary for describing what happens as the cut moves. We fix notation once and use it throughout. There are $n$ examples, each with a feature vector $\mathbf{x}_i$, a true label $y_i \in \{0,1\}$, and a score $s_i = s(\mathbf{x}_i)$ produced by some upstream model. The positive class $y=1$ is the event we want to catch: fraud, machine failure, a disease. We adopt the convention that larger scores mean more evidence for the positive class. A library score must be oriented to the selected positive label; for a two-column probability output, use that label's column. A threshold $t$ turns scores into labels through the indicator rule

$$\hat{y}_i(t) = \mathbf{1}\{ s_i \ge t \},$$

where $\mathbf{1}\{\cdot\}$ is the indicator function, equal to 1 when the statement inside the braces is true and 0 otherwise. Read the rule aloud as "predict positive when the score is at least $t$". The threshold is a single number; sweeping it produces a family of classifiers, and everything in this chapter is a way of describing that family.

At any fixed threshold, every example falls into exactly one of four cells. Those cells are not an arbitrary summary: they are the complete cross-tabulation of the true label and the decision. For a fixed sample and threshold, the four counts completely specify the unweighted binary confusion table. They do not retain score ranking, calibration or subgroup-specific behavior. Writing them as indicator sums,

$$\begin{aligned} TP(t) &= \sum_{i=1}^{n} \mathbf{1}\{ s_i \ge t \}\,\mathbf{1}\{ y_i = 1 \}, \\ FP(t) &= \sum_{i=1}^{n} \mathbf{1}\{ s_i \ge t \}\,\mathbf{1}\{ y_i = 0 \}, \\ FN(t) &= \sum_{i=1}^{n} \mathbf{1}\{ s_i < t \}\,\mathbf{1}\{ y_i = 1 \}, \\ TN(t) &= \sum_{i=1}^{n} \mathbf{1}\{ s_i < t \}\,\mathbf{1}\{ y_i = 0 \}. \end{aligned}$$

Each term in each sum is 1 exactly when the example belongs to the cell in question and 0 otherwise, so the sum simply counts the examples in that cell. $TP$ counts true positives, $FP$ false positives, $FN$ false negatives, and $TN$ true negatives, the same four categories that appear under different names in Type I and Type II Errors: a false positive is a Type I error, a false negative a Type II error.

We arrange the four counts into a matrix, with actual class labelling the rows and predicted class labelling the columns:

$$\mathbf{C}(t) = \begin{bmatrix} TN(t) & FP(t) \\ FN(t) & TP(t) \end{bmatrix}.$$

The convention matters, so we state it: row 1 means the example is truly negative, column 2 means we predicted positive. If we stack the labels into a column vector $\mathbf{y} \in \{0,1\}^n$ and the predictions into $\mathbf{u}(t) \in \{0,1\}^n$ with $u_i(t) = \mathbf{1}\{s_i \ge t\}$, the same four counts are dot products, with $\mathbf{1}$ denoting the length-$n$ vector of ones:

$$\mathbf{C}(t) = \begin{bmatrix} (\mathbf{1}-\mathbf{y})^\top (\mathbf{1}-\mathbf{u}) & (\mathbf{1}-\mathbf{y})^\top \mathbf{u} \\ \mathbf{y}^\top (\mathbf{1}-\mathbf{u}) & \mathbf{y}^\top \mathbf{u} \end{bmatrix}.$$

To see why the translation works, we expand the top-right entry: $(\mathbf{1}-\mathbf{y})^\top \mathbf{u} = \sum_i (1-y_i)u_i$. The product $(1-y_i)u_i$ is 1 only when $y_i = 0$ and $u_i = 1$, which is precisely the definition of a false positive, and it is 0 in the other three cases. A sum over an indicator product has become a single matrix multiplication, which is what lets us compute all four cells at once in vectorized code.

Two row sums are fixed by the data and do not depend on $t$. The number of actual negatives is $N_0 = TN + FP$ and the number of actual positives is $N_1 = FN + TP$, with $N_0 + N_1 = n$. The prevalence, also called the base rate, is

$$\pi = \frac{N_1}{n} = \frac{N_1}{N_0 + N_1}.$$

The column sums move with $t$: $TP + FP$ is the number of alerts raised, and $TN + FN$ is the number of examples we let pass.

Prevalence helps explain why low false-positive rates can coexist with low precision. With both classes present, we take recall $R = TP/N_1 = 0.8$ and false positive rate $\text{FPR} = FP/N_0 = 0.05$. The precision, the fraction of alerts that are real, is

$$P = \frac{TP}{TP + FP}.$$

where:

• $P$: the precision, the fraction of raised alerts that are real
• $TP$: the number of true positives at the chosen threshold
• $FP$: the number of false positives at the chosen threshold

Rewriting each count as a rate times its group size, $TP = R\,N_1$ and $FP = \text{FPR}\,N_0$, and substituting $N_1 = \pi n$ and $N_0 = (1-\pi)n$, gives

$$\begin{aligned} P &= \frac{TP}{TP + FP} \\ &= \frac{R\,N_1}{R\,N_1 + \text{FPR}\,N_0} && \text{(substitute the rate definitions)} \\ &= \frac{R\,\pi n}{R\,\pi n + \text{FPR}\,(1-\pi)\,n} && \text{(substitute } N_1 = \pi n,\; N_0 = (1-\pi)n \text{)} \\ &= \frac{R\,\pi}{R\,\pi + \text{FPR}\,(1-\pi)} && \text{(cancel } n \text{)}. \end{aligned}$$

where:

• $R$: the recall, the share of actual positives that are flagged
• $\pi$: the prevalence, the share of examples that are actually positive
• $\text{FPR}$: the false positive rate, the share of actual negatives that are wrongly flagged
• $n$: the total number of examples

Now we put in the numbers. With $\pi = 0.001$, a screening problem with one positive in a thousand,

$$P = \frac{0.8 \times 0.001}{0.8 \times 0.001 + 0.05 \times 0.999} = \frac{0.0008}{0.05075} \approx 0.0158.$$

Only 1.6 percent of alerts are real. The same classifier on a population with $\pi = 0.10$ would deliver $P = 0.08/(0.08 + 0.045) \approx 0.64$. This comparison holds recall and false positive rate fixed while changing prevalence, as under a pure prevalence shift with unchanged class-conditional score distributions. More general population changes can alter those rates too. The precision formula requires a positive alert probability.

For positive denominators, the four counts generate the following metrics, each with an operational reading. Precision is the share of alerts that are correct, recall is the share of positives we catch, and the false positive rate is the share of negatives we wrongly flag:

$$P(t) = \frac{TP(t)}{TP(t) + FP(t)}, \qquad R(t) = \frac{TP(t)}{TP(t) + FN(t)}, \qquad \text{FPR}(t) = \frac{FP(t)}{FP(t) + TN(t)}.$$

Recall is identical to the true positive rate, $\text{TPR}(t) = R(t)$, and the true negative rate $TN/(TN+FP) = 1 - \text{FPR}$ is the specificity. The harmonic mean of precision and recall is the $F_1$ score,

$$F_1(t) = \frac{2\,P(t)\,R(t)}{P(t) + R(t)},$$

where:

• $F_1(t)$: the harmonic mean of precision and recall at threshold $t$
• $P(t)$: the precision at threshold $t$
• $R(t)$: the recall at threshold $t$

Substituting the definitions of $P(t)$ and $R(t)$ and clearing denominators, when those expressions are defined, gives the equivalent count form

$$\begin{aligned} F_1(t) &= \frac{2\,\dfrac{TP}{TP+FP}\,\dfrac{TP}{TP+FN}}{\dfrac{TP}{TP+FP} + \dfrac{TP}{TP+FN}} && \text{(substitute } P, R \text{)} \\ &= \frac{2\,TP^2}{TP\,(TP+FN) + TP\,(TP+FP)} && \text{(multiply through by } (TP+FP)(TP+FN) \text{)} \\ &= \frac{2\,TP}{2\,TP + FP + FN} && \text{(cancel the common factor } TP \text{)}, \end{aligned}$$

where $TP$, $FP$, and $FN$ are evaluated at threshold $t$. The count form also has a denominator condition; reporting a value for a zero denominator requires an explicit convention. The harmonic mean punishes imbalance between the two: if $P = 1$ and $R = 0.1$, the arithmetic mean is 0.55 but $F_1 \approx 0.18$. This is why $F_1$ is described as balancing the two, though it does not encode the stated error costs. Because precision depends on prevalence, F1 can depend on prevalence too; maximizing it need not minimize operational cost.

As $t$ decreases, the prediction set grows, so $TP(t)$ and $FP(t)$ are nondecreasing while $FN(t)$ and $TN(t)$ are nonincreasing. Recall and the false positive rate are nonincreasing in $t$; precision is not monotone, because its numerator and denominator both grow as $t$ decreases and either can win.

The threshold curves plot these quantities against $t$ itself. They are the most direct picture of the tradeoff, and they are what you would show when you want to know what happens at a particular cut. The ROC and precision-recall curves are different views of the same sweep: instead of plotting against $t$, they plot one metric against another, which erases $t$ from the picture and therefore erases the information about which threshold produced each point.

Two bookkeeping conventions matter in practice and are the source of most off-by-one arguments with software. First, at $t$ above the largest score no example is predicted positive, so $TP = FP = 0$ and precision is $0/0$. For the precision-recall curve, scikit-learn appends a precision-1, recall-0 plotting endpoint. We use that convention here; it is separate from the configurable zero-division behavior of a standalone precision metric. Second, the achievable confusion matrices are indexed by the distinct score values. Two thresholds lying between the same pair of adjacent sorted scores produce identical predictions, so the curve has one point per distinct score, plus the trivial corner. Ties in the scores are handled by moving all tied examples at once.

Plotting the true positive rate against the false positive rate as $t$ sweeps from $+\infty$ down to $-\infty$ gives the receiver operating characteristic curve, which runs from $(0,0)$ at the "alert nobody" corner to $(1,1)$ at the "alert everybody" corner. The area under it is

$$\text{AUC} = \int_{0}^{1} \text{TPR}(u)\, du$$

where:

• $u$: the false positive rate coordinate, running from 0 to 1
• $\text{TPR}(u)$: the true positive rate on the ROC curve at that false positive rate

For a finite sample with both classes present, the area under the linearly connected empirical ROC points is exactly the trapezoid sum

$$\widehat{\text{AUC}} = \sum_{k} \frac{(\text{FPR}_k - \text{FPR}_{k-1})\,(\text{TPR}_k + \text{TPR}_{k-1})}{2},$$

where:

• $K$: the number of non-origin threshold points, one per distinct score group
• $k$: indexes those points from 1 to $K$ in order of nondecreasing FPR, with $\text{FPR}_0 = \text{TPR}_0 = 0$ at the origin
• $\text{FPR}_k$: the false positive rate at threshold point $k$
• $\text{TPR}_k$: the true positive rate at threshold point $k$

Each term is a trapezoid with width $\text{FPR}_k-\text{FPR}_{k-1}$ and average height $(\text{TPR}_k+\text{TPR}_{k-1})/2$. Vertical segments have zero width.

The empirical trapezoid area also has an exact pair-count interpretation. Compare every positive score with every negative score. A correctly ordered pair contributes 1, an incorrectly ordered pair 0, and a tied pair one half:

$$\widehat{\text{AUC}} = \frac{1}{N_0 N_1} \sum_{i:\,y_i=1} \sum_{j:\,y_j=0} \left[\mathbf{1}\{s_i>s_j\} + \tfrac{1}{2}\mathbf{1}\{s_i=s_j\}\right].$$

Here $i$ indexes positives, $j$ indexes negatives, and $N_0N_1$ is the number of positive-negative pairs. With distinct scores this is the fraction of correctly ordered pairs. In a tied score group with $a$ positives and $b$ negatives, the diagonal ROC segment contributes half of the $ab$ within-group pairs, which explains the half-tie term. The population interpretation is the probability that an independent positive draw outranks an independent negative draw, with half credit for ties. Scores drawn independently of labels from the same distribution in both classes have expected AUC $1/2$.

For the precision-recall curve we plot precision against recall and summarize with average precision,

$$\text{AP} = \sum_{k=1}^{K} \big(R_k - R_{k-1}\big)\, P_k,$$

where:

• $\text{AP}$: average precision, the noninterpolated summary used here
• $K$: the number of distinct score groups
• $k$: indexes the groups in order of decreasing score
• $R_k$: the recall after including the $k$-th group of tied scores, with $R_0 = 0$
• $P_k$: the precision at that point
• $R_k - R_{k-1}$: the recall gained at step $k$, the width of the staircase step

The formula is the area under a staircase: recall increases by $R_k - R_{k-1}$ at step $k$, and over that sliver of recall we use the precision achieved at the step, $P_k$. This is the definition used by scikit-learn's average_precision_score.

The trapezoidal area under the precision-recall curve is a different quantity, and the difference is not cosmetic. The trapezoid rule connects consecutive PR points with straight lines. For this finite deterministic threshold sweep, the interior points of a segment between distinct consecutive PR points are not attainable by a threshold; its endpoints are attainable. Whether the line sits above or below the AP staircase depends on the local slope, so the two summaries can differ in either direction. Here we report the noninterpolated average precision used by scikit-learn and name it explicitly. Other PR-area conventions exist, so the interpolation or summation rule should accompany a reported PR-AUC.

Both summaries have limits that decide when they are appropriate.

• ROC AUC is invariant to any strictly increasing transformation of the scores. Replacing $s$ by $s^3$, a strictly increasing sigmoid rescaling, or $\log s$ when every score is positive leaves every threshold sweep unchanged in shape, so the area is identical. The invariance is a statement that AUC measures ranking only.
• Average precision is not invariant to class prevalence. Changing $\pi$ while holding the class-conditional score distributions fixed leaves the ROC curve exactly unchanged, because both axes normalize within their own class, but it can change precision and average precision, because precision mixes the two classes. A change is not guaranteed at every point; perfect separation is a boundary case. The demonstration is in the next section.
• Neither number identifies an operating point. A model with AUC 0.9 can be deployed at a threshold that produces a 40 percent false positive rate or one that produces a 1 percent false positive rate; both are consistent with the same area.
• Neither summary uniquely rewards truthful probability values: a strictly increasing score transformation preserves both ranking summaries. AUC remains 1 after such a distortion of a perfectly separating score. Check calibration separately when the probability scale is needed for a decision.
• The full ROC area can hide behavior in a small operational region. Adding easily ranked negatives changes the comparison population and may increase AUC; it does not establish how a chosen threshold will behave. Report threshold-specific counts alongside the area.

Suppose a false positive costs $c_{FP}$ and a false negative costs $c_{FN}$, both in the same units, typically money or hours of your analyst time. We use a model with strictly positive error costs and zero cost for true positives and true negatives. Adding the same action-independent cost per example would shift every rule equally; costs that differ by action or outcome require a corresponding decision model. The empirical expected cost per example at threshold $t$ is

$$\hat{C}(t) = \frac{1}{n}\Big[ c_{FP}\,FP(t) + c_{FN}\,FN(t) \Big],$$

where:

• $\hat{C}(t)$: the observed expected cost per example at threshold $t$ (the empirical, sample-based version of the population cost below)
• $c_{FP}$: the cost of a false positive
• $c_{FN}$: the cost of a false negative
• $\text{FP}(t)$: the number of false positives at threshold $t$
• $\text{FN}(t)$: the number of false negatives at threshold $t$
• $n$: the number of examples in the sample

The factor $1/n$ converts the total cost into cost per example, matching the population expression's per-example scale. For the population version, now let $\pi=\Pr(Y=1)$ and let FPR and TPR denote population conditional rates, with both classes present. Then

$$C(t) = c_{FP}\,(1-\pi)\,\text{FPR}(t) + c_{FN}\,\pi\,\big(1 - \text{TPR}(t)\big),$$

where:

• $C(t)$: the population expected cost per example at threshold $t$
• $c_{FP}$: the cost of a false positive
• $c_{FN}$: the cost of a false negative
• $\pi = \Pr(Y=1)$: the population prevalence of the positive class
• $\text{FPR}(t)$: the population false positive rate at threshold $t$
• $\text{TPR}(t)$: the population true positive rate at threshold $t$

This expression is linear in the ROC coordinates, so level sets of constant cost are straight lines in the $(\text{FPR}, \text{TPR})$ plane. Writing $A = c_{FP}(1-\pi)$ for the coefficient of $\text{FPR}(t)$ and $B = c_{FN}\pi$ for the coefficient of $1-\text{TPR}(t)$, the cost is

$$C(t) = A\,\text{FPR}(t) + B\,\big(1 - \text{TPR}(t)\big).$$

Setting $C(t) = k$ for a fixed budget $k$ and solving for $\text{TPR}$ gives the line

$$\text{TPR} = \left(1 - \frac{k}{B}\right) + \frac{A}{B}\,\text{FPR},$$

so the lines have slope

$$\frac{A}{B} = \frac{c_{FP}\,(1-\pi)}{c_{FN}\,\pi}$$

where:

• $A = c_{FP}(1-\pi)$: the per-example weight on the false positive rate
• $B = c_{FN}\pi$: the per-example weight on the false negative rate
• $c_{FP}$: the cost of a false positive
• $c_{FN}$: the cost of a false negative
• $\pi = \Pr(Y=1)$: the population prevalence of the positive class

The slope assumes $c_{FN}\pi>0$. Setting $dC=0$ gives $c_{FP}(1-\pi)\,d\text{FPR}=c_{FN}\pi\,d\text{TPR}$. Lower cost favors smaller FPR and larger TPR; among the available threshold points, you choose the one with the smallest stated cost. Increasing false-positive cost favors fewer alerts, toward lower FPR and TPR; increasing false-negative cost favors more alerts, toward higher FPR and TPR. Ties and a discrete curve can leave the selected point unchanged.

When the score is the true conditional event probability, $s(\mathbf{x})=\Pr(Y = 1 \mid \mathbf{X} = \mathbf{x})$, the optimal threshold has a closed form. Consider a single example with score $p$, the conditional probability of a positive label. If we predict positive, we pay $c_{FP}$ when the label is 0 (probability $1-p$) and nothing when it is 1 (probability $p$), so the expected cost of a positive prediction is $c_{FP}(1-p)$. If we predict negative, we pay $c_{FN}$ when the label is 1 and nothing when it is 0, so the expected cost is $c_{FN}\,p$. We predict positive exactly when the first is no larger than the second:

$$\begin{aligned} c_{FP}(1-p) &\le c_{FN}\,p && \text{(positive no costlier than negative)} \\ c_{FP} - c_{FP}\,p &\le c_{FN}\,p && \text{(expand the left side)} \\ c_{FP} &\le c_{FN}\,p + c_{FP}\,p && \text{(add } c_{FP}\,p \text{ to both sides)} \\ c_{FP} &\le p\,\big(c_{FN} + c_{FP}\big) && \text{(factor out } p \text{)} \\ p &\ge \frac{c_{FP}}{c_{FP} + c_{FN}} && \text{(divide by the positive constant } c_{FP} + c_{FN} \text{)}, \end{aligned}$$

where $p$ is the score, i.e. the conditional probability that the example is positive. Expected total cost is the sum of conditional per-example costs, so minimizing each term yields a minimizing rule. This argument does not require the observed labels to be independent. At equality either action has the same conditional expected cost; our convention predicts positive. Writing $p^{*}=c_{FP}/(c_{FP}+c_{FN})$ for the boundary probability, its posterior odds equal the cost ratio,

$$\frac{p^{*}}{1 - p^{*}} = \frac{c_{FP}}{c_{FN}}.$$

With $c_{FP} = 5$ and $c_{FN} = 150$ the threshold is $5/155 \approx 0.032$, far below the default 0.5, and the required odds are 1 to 30. Equal error costs give the cutoff 0.5 in this model. Unequal costs give a different algebraic cutoff, although a population with no scores between the two cutoffs can yield identical decisions. Class imbalance alone does not determine the error costs.

The closed form has a precondition that is easy to miss. It requires the stated calibrated-probability assumption for the relevant population; merely lying between 0 and 1 is insufficient. When that assumption is unavailable, we drop the closed form and minimize the empirical cost $\hat{C}(t)$ over the threshold grid directly. This evaluates the available deterministic threshold states without assuming a probability scale. Its validation minimizer is an empirical choice, not a proof that it equals the population Bayes rule. What we lose is the ability to check the answer against a formula, and what we gain is freedom from the calibration assumption.

The selection protocol follows from the fact that $\hat{C}(t)$ is estimated. We need three disjoint roles. The training split fits the model that produces the scores for you to threshold. The validation split selects the threshold, by minimizing $\hat{C}(t)$ over its grid. The test split is touched once, to report the cost of the chosen rule. Selection can make the reported validation minimum optimistic in repeated samples; using test data to choose the threshold contaminates the only honest measurement we have. This is the same discipline as cross-validation, applied to a parameter that happens to be a scalar, and it is the reason hyperparameter search and nested resampling are natural companions to what we build here.

• Monotonicity. For $t_1 \le t_2$, every prediction made at $t_2$ is also made at $t_1$, so $TP(t_1) \ge TP(t_2)$ and $FP(t_1) \ge FP(t_2)$. Recall and the false positive rate are therefore nonincreasing in $t$, and the ROC curve is traced monotonically from $(0,0)$ to $(1,1)$.
• Rank invariance. If $g$ is strictly increasing, then $\mathbf{1}\{g(s_i) \ge g(t)\} = \mathbf{1}\{s_i \ge t\}$, so the entire threshold sweep, and hence the ROC curve, AUC, and the PR curve, are unchanged by replacing $s$ with $g(s)$.
• AUC as a probability. With continuous scores, $\text{AUC} = \Pr(s^{+} > s^{-})$ for independent draws from the positive and negative classes. The empirical estimate is the Mann-Whitney $U$ statistic divided by $N_0 N_1$, which is why AUC and the Wilcoxon rank-sum test are the same computation in different clothing.
• Prevalence effects. Under a pure prevalence shift with the class-conditional score distributions fixed, population ROC coordinates and AUC are unchanged. Precision and average precision can change; this need not happen under perfect separation. More general population changes can alter ROC behavior too.
• $F_1$ bounds. For defined nonnegative precision and recall with $P+R>0$, write $a = \min(P, R)$ and $b = \max(P, R)$. Then the harmonic mean satisfies $a \le 2ab/(a+b) \le \sqrt{ab} \le (a+b)/2$. The lower bound follows from $2b/(a+b) \ge 1$, so $F_1$ never falls below the smaller of the two inputs, and it never exceeds their arithmetic mean.
• Cost plateaus. Under the inclusive rule, consecutive observed scores $a<b$ give identical predictions for every $t\in(a,b]$. Empirical cost is constant on such a plateau. Distinct plateaus may tie for minimum cost, so the set of minimizing thresholds can be a union of intervals. Reporting a representative threshold does not imply its rounded value remains in the same plateau.
• Undefined corners. Precision is $0/0$ when no example is predicted positive, and recall is $0/0$ when the sample contains no positives at all. Both cases need an explicit convention, and the two curves in software differ in whether the corner point is returned.

We now build a small alerting problem where every quantity is known by construction. Each account has two features, the linear predictor is $z_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2}$, the label is drawn as $y_i \sim \text{Bernoulli}(p_i)$ with $p_i = 1/(1+e^{-z_i})$, and the model's score is exactly $p_i$, the probability you would threshold. Because the label was generated from the same probability the score reports, the score is calibrated by construction. That lets us compare a finite-sample empirical cost minimum with the population cutoff from the previous section. They need not agree exactly on a realized sample; an arbitrary score would not justify the analytic probability cutoff at all.

:::

The two ROC curves lie on top of each other and the two AUC values agree to three decimals, while the average precision values differ substantially. Both plots are honest; they answer different questions. The ROC answer is "how well does this score order the two classes", which does not depend on how the classes are mixed. The precision-recall answer is "how useful would this score be here", which very much does. If we tuned the alerting model on a sample with 20 percent positives and deployed it where positives are 2 percent, the ROC curve would have told us nothing was wrong.

The simulated data shows the machinery at scale, but its numbers arrive from code. A tiny hand-worked case shows where each number comes from and makes the formulas concrete. Sixteen accounts have been scored, four of them are fraudulent, and the scores are sorted from highest to lowest. Every number below is small enough for you to check with a calculator.

There are $N_1 = 4$ positives and $N_0 = 12$ negatives, so the prevalence is $\pi = 4/16 = 0.25$. The costs are $c_{FP} = 5$ dollars for reviewing an account that turns out to be clean and $c_{FN} = 150$ dollars for missing a fraudulent one. In other words, each false positive costs 5 dollars and each false negative costs 150 dollars.

Start at the default threshold $t = 0.50$. The rule flags the first six accounts, since their scores are at least 0.50. Of those six, three are fraudulent (ranks 1, 2, 4) and three are not (ranks 3, 5, 6). The one remaining fraud sits at rank 7 with a score of 0.45, so it is missed. These four counts are exactly the four cells of the confusion matrix, and they are complete: every account appears once.

$$\mathbf{C}(0.50) = \begin{bmatrix} TN & FP \\ FN & TP \end{bmatrix} = \begin{bmatrix} 9 & 3 \\ 1 & 3 \end{bmatrix}.$$

The metrics follow mechanically. Precision is $3/(3+3) = 0.500$, recall is $3/(3+1) = 0.750$, the false positive rate is $3/(3+9) = 0.250$, and $F_1 = 2 \times 0.5 \times 0.75/(0.5 + 0.75) = 0.600$, which the count form confirms: $2 \times 3/(2 \times 3 + 3 + 1) = 6/10$. The expected cost per account is

$$\hat{C}(0.50) = \frac{5 \times 3 + 150 \times 1}{16} = \frac{165}{16} = 10.3125.$$

Now lower the threshold to 0.45, which adds rank 7 to the alert queue. Nothing else changes, because no other score lies between 0.45 and 0.50. The confusion matrix becomes

$$\mathbf{C}(0.45) = \begin{bmatrix} 9 & 3 \\ 0 & 4 \end{bmatrix},$$

so precision rises to $4/7 \approx 0.571$, recall rises to $4/4 = 1.000$, the false positive rate is unchanged at $3/12 = 0.250$, and $F_1 = 2 \times 4/(2 \times 4 + 3 + 0) = 8/11 \approx 0.727$.

where:

• $TP = 4$: the number of flagged frauds after including rank 7
• $FP = 3$: the number of flagged clean accounts (ranks 3, 5, 6)
• $FN = 0$: no fraud remains unflagged
• $TN = 9$: the number of unflagged clean accounts

The cost drops to

$$\hat{C}(0.45) = \frac{5 \times 3 + 150 \times 0}{16} = \frac{15}{16} = 0.9375.$$

Recall rises and precision also rises here because the one added alert is a true positive, which is the opposite of the general tendency: we noted earlier that raising the threshold from 0.45 to 0.50 reduced precision from 4/7 to 1/2 while reducing recall. When moving toward lower thresholds, adding a true positive can lift both metrics, while adding a false positive leaves recall unchanged and lowers precision when the queue already contains a true positive. This example shows why precision need not move in the opposite direction from recall. The cost, however, falls by more than a factor of ten. Multiplying this observed difference by a hypothetical 10,000 accounts gives $93,750. Interpreting that as a future daily saving would additionally require the example error rates and stated costs to transfer to the future stream.

The threshold need not be exactly 0.45. Any value in the interval between the seventh and eighth scores, $t \in (0.38, 0.45]$, produces the same predictions and therefore the same cost. This is the cost plateau of the previous section, and it means the reported threshold should be understood as a representative of an interval. The midpoint rule used in the code would report 0.415.

Next we check the curve summaries against the same sixteen points. Ordering the positives by rank, the true positive count after each is $1, 2, 3, 4$ at ranks $1, 2, 4, 7$, so recall takes the values 0.25, 0.50, 0.75, and 1.00 with precisions 1/1, 2/2, 3/4, and 4/7. Average precision sums the recall increments weighted by the precision achieved:

$$\begin{aligned} \text{AP} &= \tfrac14\!\left(1+1+\tfrac34+\tfrac47\right) \\ &= \tfrac12+\tfrac{3}{16}+\tfrac17 \\ &= \tfrac{93}{112} \approx 0.8304. \end{aligned}$$

For the AUC we count, for each positive, how many negatives it outranks. Rank 1 outranks all twelve, rank 2 also outranks all twelve, rank 4 outranks the eleven negatives below it, and rank 7 outranks the nine below it:

$$\widehat{\text{AUC}} = \frac{12 + 12 + 11 + 9}{12 \times 4} = \frac{44}{48} \approx 0.9167.$$

The trapezoidal area under the same precision-recall points is a different number. The horizontal segments contribute $0.25 \times 1$ and $0.25 \times 1$, the segment from recall 0.50 to 0.75 connects precision 2/3 to 3/4, and the segment from 0.75 to 1.00 connects precision 1/2 to 4/7:

$$\widehat{\text{PR}}_{\text{trap}} = 0.25 + 0.25 + 0.25\!\left(\tfrac{2/3 + 3/4}{2}\right) + 0.25\!\left(\tfrac{1/2 + 4/7}{2}\right) \approx 0.8110.$$

The two do not agree in this case; the trapezoid differs from average precision. Here average precision is about 0.8304 and the trapezoidal area about 0.8110, a difference of $13/672\approx0.0193$. The definitions can sometimes yield the same number, even when precision is not constant, because segment differences can cancel. Name the convention rather than treating the two definitions as interchangeable.

Finally, the closed form for calibrated scores gives $t^{*} = 5/(5+150) \approx 0.0323$. Applying it here would flag every account scoring at least 0.0323, which is ranks 1 through 15, giving $TP = 4$ and $FP = 11$ for a cost of $(5 \times 11 + 150 \times 0)/16 = 3.4375$. The toy scores are used here as an arbitrary ranking, so we have not established the conditional-probability assumption for that cutoff. Its observed cost is larger than the empirical optimum of 0.9375. That finite-sample comparison alone would not prove miscalibration: a sample-selected rule can also beat a population-optimal rule on the sample used to select it.

We close the example by checking the hand arithmetic against the code, which is the habit that keeps your worked examples honest.

We now run the full protocol on the simulated alerting data, using the toolkit from the visualization section. The pipeline has three roles and we keep them strictly separate: a training split would fit the scorer, the validation split selects the threshold, and the test split reports. Since the scores are given, only the last two appear in the code.

The comparison we want is between rules, not models. We evaluate three decision rules: the default 0.50 cut, the closed-form threshold that would be correct if the scores were calibrated, and the threshold selected by minimizing empirical cost on validation. Each fixed rule is applied to both splits. Their realized costs can be compared, but one validation/test difference does not identify the expected selection bias.

• The threshold itself is the decision variable. It is selected on data that plays no part in fitting the scorer and is fixed before any test evaluation.
• c_fp and c_fn are the two costs, and they must be in the same units. Their ratio, not their absolute size, determines the threshold that a calibrated score would use, so the closed form is $c_{FP}/(c_{FP}+c_{FN})$ and the corresponding odds are $c_{FP}/c_{FN}$.
• The class prevalence affects precision and average precision. Transporting them to a different prevalence requires assumptions about the class-conditional score distributions. The closed-form probability cutoff instead depends on the error-cost ratio; the probabilities to which it is applied must represent the deployment population.
• The calibration status of the score decides whether the closed form is admissible. A rank-only score supports empirical cost minimization but not the analytic rule.
• Uncertainty depends on the metric denominator and the sampling and selection design. Recall uses actual positives as its denominator; precision uses predicted positives, or alerts.

• confusion_counts and binary_metrics give the four counts and the derived metrics at a single threshold. They are the primitives; everything else is a loop or a vectorization over them.
• threshold_sweep computes the whole family of confusion matrices in one pass by sorting the scores once and taking cumulative sums. Its cost is $O(n \log n)$, dominated by the sort.
• curve_points, roc_auc, average_precision and pr_trapezoid turn the sweep into curves and summary numbers. Keeping average_precision and pr_trapezoid as separate functions is deliberate, because they are separate quantities.
• cost_curve and select_threshold implement the decision rule. select_threshold uses a midpoint for an interior optimal plateau when it is representable, falling back to the inclusive upper score if rounding reaches the excluded lower score. It reports infinity for the no-alert state and a valid representative between zero and the minimum score for the all-alert state. These reproducible conventions do not guarantee greater sampling stability than an endpoint.
• The scikit-learn counterparts are roc_auc_score, average_precision_score, precision_recall_curve, roc_curve and confusion_matrix. Recent versions also provide TunedThresholdClassifierCV, which wraps the same select-on-validation logic in a cross-validated estimator; the primitive version here is easier to audit and to adapt to asymmetric costs.

Metrics serve two different jobs: comparing scores before a deployment decision and operating a score after one. For comparison, ROC AUC is a reasonable default when both classes are represented and the question is ordering; average precision is often more informative when positives are rare and false alerts compete for finite review capacity. For deployment, neither summary is the deliverable. The deliverable is a fixed threshold, the confusion matrix it produces on held-out data, and a statement of the cost or service level implied by that operating point. A model comparison can end with an AUC; an operational decision cannot.

Cost elicitation belongs to the modeling work rather than to a final sign-off. A false positive may consume analyst time, create customer friction, or trigger unnecessary account freezes; a false negative may produce chargebacks, missed maintenance, or regulatory exposure. Estimate these components where evidence permits and record the uncertainty in the ratio. When no credible ratio can be assembled, present recall, precision, false positive rate and alert volume over the thresholds. Cost curves can still be shown for explicitly assumed ratios, but cannot be specified without costs. Choosing an operating point expresses an operational tradeoff even when that tradeoff has not been fully priced. The same discussion should establish whether one threshold is defensible across segments. Different error-cost ratios can justify different probability cutoffs. Different base rates can change the appropriate cutoff on a shared raw score or require segment-aware probability estimation. If scores already represent the relevant segment-conditional probabilities and error costs are common, the same probability cutoff remains optimal across segments.

Keep the roles of training, validation, and test data distinct. The scorer is fitted on training data, the threshold is selected on validation data by minimizing observed cost or by applying a justified closed-form cutoff, and the test split is opened once to report the resulting rule. When the score is a calibrated probability, you should compare the analytic cutoff with the empirical minimizer; disagreement is information about calibration or finite-sample variability, not a reason to force the two numbers together. When the score is only a ranking, skip the closed form and minimize empirical cost directly.

Report the decision as an interval and a cost, not as a suspiciously precise threshold. Because predictions change only when a score is crossed, the optimal threshold is usually a plateau; the midpoint is a convenient representative, but the endpoints and the cost across the plateau are more informative than three decimal places. Bootstrap the selection procedure to show how far the threshold moves under resampling, and, when estimating cost uncertainty, report it separately because the two quantities have different units. The illustration above used 300 resamples; more resamples can reduce Monte Carlo noise in this resampling summary, but cannot repair limited validation data or establish interval coverage by themselves.

Alongside the chosen threshold, report recall, precision, false positive rate, alert volume, and the confusion matrix on held-out data. State the positive class and the score orientation explicitly, since reversing the score reverses its ranking and requires changing the decision convention. If average precision is quoted, quote the prevalence of the evaluation set with it; if ROC AUC is quoted, make clear that it is a ranking summary and not evidence that the deployed alert volume is acceptable.

The minimum inputs are paired labels and scores for validation and test data, with the positive class oriented so that larger scores mean stronger evidence for the event of interest. Both classes must be present for ROC AUC; the stated average-precision definition requires at least one positive, and at least one positive prediction must occur at a threshold for precision to be defined without an explicit convention. Costs should be supplied in the same units, and any segment identifiers needed for subgroup reporting should be retained. For the closed-form probability cutoff, the scores must be probabilities for the deployment population; for empirical cost selection, arbitrary finite scores are acceptable.

Little preprocessing is required for the rank-based summaries because ROC AUC and average precision are invariant to strictly increasing transformations of the score. Rescalings that are defined and strictly increasing on the score domain do not change its ordering; a logarithm, for example, requires positive scores, but they do change the numeric threshold, so the threshold must be expressed on the same scale and orientation as the scores that will be seen in deployment. Ties should be handled as tied groups rather than broken arbitrarily. Missing scores need an explicit policy, and a calibration layer, if used, is a separate fitted component with its own validation requirements.

Prevalence deserves its own check. Under a pure prevalence shift that holds both class-conditional score distributions fixed, the identity $P = R\pi/(R\pi + \text{FPR}(1-\pi))$ translates recall and false positive rate into deployment precision when the alert probability is positive. Label-only enrichment of a validation set can create such a shift. More general population changes can alter the class-conditional distributions and therefore the ROC curve itself, so changing $\pi$ in the formula alone is not a substitute for re-evaluating on a representative sample.

A damaging mistake is selecting the threshold on the test set and then reporting performance on the same observations. That is leakage with extra steps: the reported cost is optimized on the data used to judge it, which can bias the reported cost downward and removes an independent evaluation of the choice. This selection effect is a repeated-sample statement, not a guaranteed ordering for every realized result. The fix is procedural rather than mathematical: record the validation-selected threshold before opening the test split, and do not revise it after seeing test results.

A second recurring mistake you should avoid is treating $F_1$ as if it were a decision rule. Maximizing $F_1$ encodes an implicit and usually unstated tradeoff between precision and recall, and it does not use the stated error costs. It is entirely possible for the $F_1$-optimal threshold to sit far from the cost-optimal one, especially when prevalence or the cost ratio is extreme. If a stakeholder requests the $F_1$-optimal threshold, show it beside the cost-optimal rule and name the difference in expected cost.

Other pitfalls follow from using the wrong summary for the wrong job. A high ROC AUC can coexist with an unusable number of false alerts at the operating point, and average precision estimated at one prevalence cannot be assumed to transfer unchanged to another. Applying the closed form $c_{FP}/(c_{FP}+c_{FN})$ to an uncalibrated margin or vote fraction produces a threshold that looks principled but has no probability interpretation. Reporting a threshold to many decimal places when the cost curve is flat over a wide interval overstates the precision of the selection, and ignoring segment-level error rates can hide a rule that is acceptable on average and harmful to a subgroup.

The threshold sweep requires one sort followed by linear scans; its runtime depends on the sample size and implementation. Sorting the scores once costs $O(n \log n)$, cumulative sums produce all distinct confusion matrices in $O(n)$, and the ROC curve, precision-recall curve, and cost curve can all reuse the same sweep. Memory is linear in the sample size, dominated by the sorted scores and the cumulative count arrays. For a single validation or test set, this is practical for millions of observations on a modern machine, and the dominant cost is usually the sort rather than the metric arithmetic.

Repeated resampling multiplies that cost. A bootstrap with $B$ resamples performs $B$ sorts and sweeps, so the total work is $O(B\,n \log n)$; the 300-resample illustration is modest, while thousands of resamples on very large data can become the bottleneck. Reuse curve arrays across plotting and reporting cells, avoid recomputing the sweep for each figure, and parallelize resamples when the implementation allows. Calibration adds a separate fitting cost and should be validated as its own step rather than folded silently into threshold selection.

Track alert volume and, when outcomes are available, precision and confusion counts at the fixed threshold. Precision and the confirmation rate among alerts describe the same fraction; estimate it from complete or representative alert outcomes. False-negative counts and recall also need outcomes for unflagged cases, so reviewed alerts alone cannot supply a full confusion matrix. ROC AUC can complement these operational quantities, but needs both labeled classes and can stay unchanged under a pure prevalence shift. Segment-level summaries can expose differences hidden by aggregate results.

The threshold should be revisited when error costs, prevalence, or the class-conditional score distribution change enough to move the operating point. A change in prevalence alone can leave recall and false positive rate unchanged under a pure shift while moving precision, so alert capacity and review workload should be monitored separately from ranking quality. If the score is used as a probability, check its calibration for the target population and recalibrate if needed before applying the closed-form cutoff; if it is used only as a ranking, rerun empirical cost selection on a representative validation sample. The honest deployment report includes the chosen threshold or plateau, the held-out cost, the alert volume, and the uncertainty from the selection procedure.

A score is not a decision. Turning a score into a decision requires a threshold, and the threshold is chosen by trading false positives against false negatives. The confusion matrix records that trade at one point; precision, recall, and the false positive rate read the trade off the matrix; the threshold curves, ROC curve, and precision-recall curve show the whole family of trades available from a fixed ranking.

The summary numbers answer narrow questions. ROC AUC compares independent positive and negative draws, awarding half credit for ties. It is invariant to strictly increasing score transformations and is unchanged by a prevalence-only shift that preserves both class-conditional score distributions. Average precision concentrates on the positive class, summarizes the precision-recall curve using the stated noninterpolated convention, and can depend on prevalence, so it cannot be assumed to transfer unchanged between populations. Neither number identifies an operating point, and neither establishes calibration. The trapezoidal area under a precision-recall curve uses a different summation convention from average precision; state which one is reported.

The threshold itself is an optimization problem once costs are named. Minimizing expected cost gives the population objective $c_{FP}(1-\pi)\text{FPR} + c_{FN}\pi(1-\text{TPR})$, whose level sets are straight lines in ROC space with slope $c_{FP}(1-\pi)/(c_{FN}\pi)$. When the score is a calibrated probability, the optimum collapses to the closed form $c_{FP}/(c_{FP}+c_{FN})$, or equivalently to posterior odds equal to the cost ratio. When that probability assumption is unavailable, sweeping validation cost still selects an empirical threshold for the given ranking; it does not certify the same population optimum.

The discipline that makes the resulting number trustworthy is the split discipline. Threshold selection is a fitting step, so it consumes validation data; the test split is touched once, to report. The selected threshold is typically not unique, since the cost is a step function with plateaus, and thresholds can vary across resamples while their observed costs remain similar. Threshold and cost dispersion have different units and should be reported on their own scales; these observations alone do not guarantee future-risk stability. Reporting a range and a cost, rather than three decimal places, is the honest summary, and it is what a team needs in order to know whether the decision is stable enough to automate.