Classification Metrics and Decision Thresholds › Formula

A fitted classifier has not yet made a decision. It has assigned each example a score, and those scores induce an ordering. To act on that ordering we choose a cut point in the ranked list and alert on everything above it. That cut point is the threshold, and the formulas below are the vocabulary for describing what happens as the cut moves. We fix notation once and use it throughout. There are $n$ examples, each with a feature vector $\mathbf{x}_i$, a true label $y_i \in \{0,1\}$, and a score $s_i = s(\mathbf{x}_i)$ produced by some upstream model. The positive class $y=1$ is the event we want to catch: fraud, machine failure, a disease. We adopt the convention that larger scores mean more evidence for the positive class. A library score must be oriented to the selected positive label; for a two-column probability output, use that label's column. A threshold $t$ turns scores into labels through the indicator rule

$$\hat{y}_i(t) = \mathbf{1}\{ s_i \ge t \},$$

where $\mathbf{1}\{\cdot\}$ is the indicator function, equal to 1 when the statement inside the braces is true and 0 otherwise. Read the rule aloud as "predict positive when the score is at least $t$". The threshold is a single number; sweeping it produces a family of classifiers, and everything in this chapter is a way of describing that family.

At any fixed threshold, every example falls into exactly one of four cells. Those cells are not an arbitrary summary: they are the complete cross-tabulation of the true label and the decision. For a fixed sample and threshold, the four counts completely specify the unweighted binary confusion table. They do not retain score ranking, calibration or subgroup-specific behavior. Writing them as indicator sums,

$$\begin{aligned} TP(t) &= \sum_{i=1}^{n} \mathbf{1}\{ s_i \ge t \}\,\mathbf{1}\{ y_i = 1 \}, \\ FP(t) &= \sum_{i=1}^{n} \mathbf{1}\{ s_i \ge t \}\,\mathbf{1}\{ y_i = 0 \}, \\ FN(t) &= \sum_{i=1}^{n} \mathbf{1}\{ s_i < t \}\,\mathbf{1}\{ y_i = 1 \}, \\ TN(t) &= \sum_{i=1}^{n} \mathbf{1}\{ s_i < t \}\,\mathbf{1}\{ y_i = 0 \}. \end{aligned}$$

Each term in each sum is 1 exactly when the example belongs to the cell in question and 0 otherwise, so the sum simply counts the examples in that cell. $TP$ counts true positives, $FP$ false positives, $FN$ false negatives, and $TN$ true negatives, the same four categories that appear under different names in Type I and Type II Errors: a false positive is a Type I error, a false negative a Type II error.

We arrange the four counts into a matrix, with actual class labelling the rows and predicted class labelling the columns:

$$\mathbf{C}(t) = \begin{bmatrix} TN(t) & FP(t) \\ FN(t) & TP(t) \end{bmatrix}.$$

The convention matters, so we state it: row 1 means the example is truly negative, column 2 means we predicted positive. If we stack the labels into a column vector $\mathbf{y} \in \{0,1\}^n$ and the predictions into $\mathbf{u}(t) \in \{0,1\}^n$ with $u_i(t) = \mathbf{1}\{s_i \ge t\}$, the same four counts are dot products, with $\mathbf{1}$ denoting the length-$n$ vector of ones:

$$\mathbf{C}(t) = \begin{bmatrix} (\mathbf{1}-\mathbf{y})^\top (\mathbf{1}-\mathbf{u}) & (\mathbf{1}-\mathbf{y})^\top \mathbf{u} \\ \mathbf{y}^\top (\mathbf{1}-\mathbf{u}) & \mathbf{y}^\top \mathbf{u} \end{bmatrix}.$$

To see why the translation works, we expand the top-right entry: $(\mathbf{1}-\mathbf{y})^\top \mathbf{u} = \sum_i (1-y_i)u_i$. The product $(1-y_i)u_i$ is 1 only when $y_i = 0$ and $u_i = 1$, which is precisely the definition of a false positive, and it is 0 in the other three cases. A sum over an indicator product has become a single matrix multiplication, which is what lets us compute all four cells at once in vectorized code.

Two row sums are fixed by the data and do not depend on $t$. The number of actual negatives is $N_0 = TN + FP$ and the number of actual positives is $N_1 = FN + TP$, with $N_0 + N_1 = n$. The prevalence, also called the base rate, is

$$\pi = \frac{N_1}{n} = \frac{N_1}{N_0 + N_1}.$$

The column sums move with $t$: $TP + FP$ is the number of alerts raised, and $TN + FN$ is the number of examples we let pass.

Prevalence helps explain why low false-positive rates can coexist with low precision. With both classes present, we take recall $R = TP/N_1 = 0.8$ and false positive rate $\text{FPR} = FP/N_0 = 0.05$. The precision, the fraction of alerts that are real, is

$$P = \frac{TP}{TP + FP}.$$

where:

• $P$: the precision, the fraction of raised alerts that are real
• $TP$: the number of true positives at the chosen threshold
• $FP$: the number of false positives at the chosen threshold

Rewriting each count as a rate times its group size, $TP = R\,N_1$ and $FP = \text{FPR}\,N_0$, and substituting $N_1 = \pi n$ and $N_0 = (1-\pi)n$, gives

$$\begin{aligned} P &= \frac{TP}{TP + FP} \\ &= \frac{R\,N_1}{R\,N_1 + \text{FPR}\,N_0} && \text{(substitute the rate definitions)} \\ &= \frac{R\,\pi n}{R\,\pi n + \text{FPR}\,(1-\pi)\,n} && \text{(substitute } N_1 = \pi n,\; N_0 = (1-\pi)n \text{)} \\ &= \frac{R\,\pi}{R\,\pi + \text{FPR}\,(1-\pi)} && \text{(cancel } n \text{)}. \end{aligned}$$

where:

• $R$: the recall, the share of actual positives that are flagged
• $\pi$: the prevalence, the share of examples that are actually positive
• $\text{FPR}$: the false positive rate, the share of actual negatives that are wrongly flagged
• $n$: the total number of examples

Now we put in the numbers. With $\pi = 0.001$, a screening problem with one positive in a thousand,

$$P = \frac{0.8 \times 0.001}{0.8 \times 0.001 + 0.05 \times 0.999} = \frac{0.0008}{0.05075} \approx 0.0158.$$

Only 1.6 percent of alerts are real. The same classifier on a population with $\pi = 0.10$ would deliver $P = 0.08/(0.08 + 0.045) \approx 0.64$. This comparison holds recall and false positive rate fixed while changing prevalence, as under a pure prevalence shift with unchanged class-conditional score distributions. More general population changes can alter those rates too. The precision formula requires a positive alert probability.

For positive denominators, the four counts generate the following metrics, each with an operational reading. Precision is the share of alerts that are correct, recall is the share of positives we catch, and the false positive rate is the share of negatives we wrongly flag:

$$P(t) = \frac{TP(t)}{TP(t) + FP(t)}, \qquad R(t) = \frac{TP(t)}{TP(t) + FN(t)}, \qquad \text{FPR}(t) = \frac{FP(t)}{FP(t) + TN(t)}.$$

Recall is identical to the true positive rate, $\text{TPR}(t) = R(t)$, and the true negative rate $TN/(TN+FP) = 1 - \text{FPR}$ is the specificity. The harmonic mean of precision and recall is the $F_1$ score,

$$F_1(t) = \frac{2\,P(t)\,R(t)}{P(t) + R(t)},$$

where:

• $F_1(t)$: the harmonic mean of precision and recall at threshold $t$
• $P(t)$: the precision at threshold $t$
• $R(t)$: the recall at threshold $t$

Substituting the definitions of $P(t)$ and $R(t)$ and clearing denominators, when those expressions are defined, gives the equivalent count form

$$\begin{aligned} F_1(t) &= \frac{2\,\dfrac{TP}{TP+FP}\,\dfrac{TP}{TP+FN}}{\dfrac{TP}{TP+FP} + \dfrac{TP}{TP+FN}} && \text{(substitute } P, R \text{)} \\ &= \frac{2\,TP^2}{TP\,(TP+FN) + TP\,(TP+FP)} && \text{(multiply through by } (TP+FP)(TP+FN) \text{)} \\ &= \frac{2\,TP}{2\,TP + FP + FN} && \text{(cancel the common factor } TP \text{)}, \end{aligned}$$

where $TP$, $FP$, and $FN$ are evaluated at threshold $t$. The count form also has a denominator condition; reporting a value for a zero denominator requires an explicit convention. The harmonic mean punishes imbalance between the two: if $P = 1$ and $R = 0.1$, the arithmetic mean is 0.55 but $F_1 \approx 0.18$. This is why $F_1$ is described as balancing the two, though it does not encode the stated error costs. Because precision depends on prevalence, F1 can depend on prevalence too; maximizing it need not minimize operational cost.

As $t$ decreases, the prediction set grows, so $TP(t)$ and $FP(t)$ are nondecreasing while $FN(t)$ and $TN(t)$ are nonincreasing. Recall and the false positive rate are nonincreasing in $t$; precision is not monotone, because its numerator and denominator both grow as $t$ decreases and either can win.

The threshold curves plot these quantities against $t$ itself. They are the most direct picture of the tradeoff, and they are what you would show when you want to know what happens at a particular cut. The ROC and precision-recall curves are different views of the same sweep: instead of plotting against $t$, they plot one metric against another, which erases $t$ from the picture and therefore erases the information about which threshold produced each point.

Two bookkeeping conventions matter in practice and are the source of most off-by-one arguments with software. First, at $t$ above the largest score no example is predicted positive, so $TP = FP = 0$ and precision is $0/0$. For the precision-recall curve, scikit-learn appends a precision-1, recall-0 plotting endpoint. We use that convention here; it is separate from the configurable zero-division behavior of a standalone precision metric. Second, the achievable confusion matrices are indexed by the distinct score values. Two thresholds lying between the same pair of adjacent sorted scores produce identical predictions, so the curve has one point per distinct score, plus the trivial corner. Ties in the scores are handled by moving all tied examples at once.

Plotting the true positive rate against the false positive rate as $t$ sweeps from $+\infty$ down to $-\infty$ gives the receiver operating characteristic curve, which runs from $(0,0)$ at the "alert nobody" corner to $(1,1)$ at the "alert everybody" corner. The area under it is

$$\text{AUC} = \int_{0}^{1} \text{TPR}(u)\, du$$

where:

• $u$: the false positive rate coordinate, running from 0 to 1
• $\text{TPR}(u)$: the true positive rate on the ROC curve at that false positive rate

For a finite sample with both classes present, the area under the linearly connected empirical ROC points is exactly the trapezoid sum

$$\widehat{\text{AUC}} = \sum_{k} \frac{(\text{FPR}_k - \text{FPR}_{k-1})\,(\text{TPR}_k + \text{TPR}_{k-1})}{2},$$

where:

• $K$: the number of non-origin threshold points, one per distinct score group
• $k$: indexes those points from 1 to $K$ in order of nondecreasing FPR, with $\text{FPR}_0 = \text{TPR}_0 = 0$ at the origin
• $\text{FPR}_k$: the false positive rate at threshold point $k$
• $\text{TPR}_k$: the true positive rate at threshold point $k$

Each term is a trapezoid with width $\text{FPR}_k-\text{FPR}_{k-1}$ and average height $(\text{TPR}_k+\text{TPR}_{k-1})/2$. Vertical segments have zero width.

The empirical trapezoid area also has an exact pair-count interpretation. Compare every positive score with every negative score. A correctly ordered pair contributes 1, an incorrectly ordered pair 0, and a tied pair one half:

$$\widehat{\text{AUC}} = \frac{1}{N_0 N_1} \sum_{i:\,y_i=1} \sum_{j:\,y_j=0} \left[\mathbf{1}\{s_i>s_j\} + \tfrac{1}{2}\mathbf{1}\{s_i=s_j\}\right].$$

Here $i$ indexes positives, $j$ indexes negatives, and $N_0N_1$ is the number of positive-negative pairs. With distinct scores this is the fraction of correctly ordered pairs. In a tied score group with $a$ positives and $b$ negatives, the diagonal ROC segment contributes half of the $ab$ within-group pairs, which explains the half-tie term. The population interpretation is the probability that an independent positive draw outranks an independent negative draw, with half credit for ties. Scores drawn independently of labels from the same distribution in both classes have expected AUC $1/2$.

For the precision-recall curve we plot precision against recall and summarize with average precision,

$$\text{AP} = \sum_{k=1}^{K} \big(R_k - R_{k-1}\big)\, P_k,$$

where:

• $\text{AP}$: average precision, the noninterpolated summary used here
• $K$: the number of distinct score groups
• $k$: indexes the groups in order of decreasing score
• $R_k$: the recall after including the $k$-th group of tied scores, with $R_0 = 0$
• $P_k$: the precision at that point
• $R_k - R_{k-1}$: the recall gained at step $k$, the width of the staircase step

The formula is the area under a staircase: recall increases by $R_k - R_{k-1}$ at step $k$, and over that sliver of recall we use the precision achieved at the step, $P_k$. This is the definition used by scikit-learn's average_precision_score.

The trapezoidal area under the precision-recall curve is a different quantity, and the difference is not cosmetic. The trapezoid rule connects consecutive PR points with straight lines. For this finite deterministic threshold sweep, the interior points of a segment between distinct consecutive PR points are not attainable by a threshold; its endpoints are attainable. Whether the line sits above or below the AP staircase depends on the local slope, so the two summaries can differ in either direction. Here we report the noninterpolated average precision used by scikit-learn and name it explicitly. Other PR-area conventions exist, so the interpolation or summation rule should accompany a reported PR-AUC.

Both summaries have limits that decide when they are appropriate.

• ROC AUC is invariant to any strictly increasing transformation of the scores. Replacing $s$ by $s^3$, a strictly increasing sigmoid rescaling, or $\log s$ when every score is positive leaves every threshold sweep unchanged in shape, so the area is identical. The invariance is a statement that AUC measures ranking only.
• Average precision is not invariant to class prevalence. Changing $\pi$ while holding the class-conditional score distributions fixed leaves the ROC curve exactly unchanged, because both axes normalize within their own class, but it can change precision and average precision, because precision mixes the two classes. A change is not guaranteed at every point; perfect separation is a boundary case. The demonstration is in the next section.
• Neither number identifies an operating point. A model with AUC 0.9 can be deployed at a threshold that produces a 40 percent false positive rate or one that produces a 1 percent false positive rate; both are consistent with the same area.
• Neither summary uniquely rewards truthful probability values: a strictly increasing score transformation preserves both ranking summaries. AUC remains 1 after such a distortion of a perfectly separating score. Check calibration separately when the probability scale is needed for a decision.
• The full ROC area can hide behavior in a small operational region. Adding easily ranked negatives changes the comparison population and may increase AUC; it does not establish how a chosen threshold will behave. Report threshold-specific counts alongside the area.

Suppose a false positive costs $c_{FP}$ and a false negative costs $c_{FN}$, both in the same units, typically money or hours of your analyst time. We use a model with strictly positive error costs and zero cost for true positives and true negatives. Adding the same action-independent cost per example would shift every rule equally; costs that differ by action or outcome require a corresponding decision model. The empirical expected cost per example at threshold $t$ is

$$\hat{C}(t) = \frac{1}{n}\Big[ c_{FP}\,FP(t) + c_{FN}\,FN(t) \Big],$$

where:

• $\hat{C}(t)$: the observed expected cost per example at threshold $t$ (the empirical, sample-based version of the population cost below)
• $c_{FP}$: the cost of a false positive
• $c_{FN}$: the cost of a false negative
• $\text{FP}(t)$: the number of false positives at threshold $t$
• $\text{FN}(t)$: the number of false negatives at threshold $t$
• $n$: the number of examples in the sample

The factor $1/n$ converts the total cost into cost per example, matching the population expression's per-example scale. For the population version, now let $\pi=\Pr(Y=1)$ and let FPR and TPR denote population conditional rates, with both classes present. Then

$$C(t) = c_{FP}\,(1-\pi)\,\text{FPR}(t) + c_{FN}\,\pi\,\big(1 - \text{TPR}(t)\big),$$

where:

• $C(t)$: the population expected cost per example at threshold $t$
• $c_{FP}$: the cost of a false positive
• $c_{FN}$: the cost of a false negative
• $\pi = \Pr(Y=1)$: the population prevalence of the positive class
• $\text{FPR}(t)$: the population false positive rate at threshold $t$
• $\text{TPR}(t)$: the population true positive rate at threshold $t$

This expression is linear in the ROC coordinates, so level sets of constant cost are straight lines in the $(\text{FPR}, \text{TPR})$ plane. Writing $A = c_{FP}(1-\pi)$ for the coefficient of $\text{FPR}(t)$ and $B = c_{FN}\pi$ for the coefficient of $1-\text{TPR}(t)$, the cost is

$$C(t) = A\,\text{FPR}(t) + B\,\big(1 - \text{TPR}(t)\big).$$

Setting $C(t) = k$ for a fixed budget $k$ and solving for $\text{TPR}$ gives the line

$$\text{TPR} = \left(1 - \frac{k}{B}\right) + \frac{A}{B}\,\text{FPR},$$

so the lines have slope

$$\frac{A}{B} = \frac{c_{FP}\,(1-\pi)}{c_{FN}\,\pi}$$

where:

• $A = c_{FP}(1-\pi)$: the per-example weight on the false positive rate
• $B = c_{FN}\pi$: the per-example weight on the false negative rate
• $c_{FP}$: the cost of a false positive
• $c_{FN}$: the cost of a false negative
• $\pi = \Pr(Y=1)$: the population prevalence of the positive class

The slope assumes $c_{FN}\pi>0$. Setting $dC=0$ gives $c_{FP}(1-\pi)\,d\text{FPR}=c_{FN}\pi\,d\text{TPR}$. Lower cost favors smaller FPR and larger TPR; among the available threshold points, you choose the one with the smallest stated cost. Increasing false-positive cost favors fewer alerts, toward lower FPR and TPR; increasing false-negative cost favors more alerts, toward higher FPR and TPR. Ties and a discrete curve can leave the selected point unchanged.

When the score is the true conditional event probability, $s(\mathbf{x})=\Pr(Y = 1 \mid \mathbf{X} = \mathbf{x})$, the optimal threshold has a closed form. Consider a single example with score $p$, the conditional probability of a positive label. If we predict positive, we pay $c_{FP}$ when the label is 0 (probability $1-p$) and nothing when it is 1 (probability $p$), so the expected cost of a positive prediction is $c_{FP}(1-p)$. If we predict negative, we pay $c_{FN}$ when the label is 1 and nothing when it is 0, so the expected cost is $c_{FN}\,p$. We predict positive exactly when the first is no larger than the second:

$$\begin{aligned} c_{FP}(1-p) &\le c_{FN}\,p && \text{(positive no costlier than negative)} \\ c_{FP} - c_{FP}\,p &\le c_{FN}\,p && \text{(expand the left side)} \\ c_{FP} &\le c_{FN}\,p + c_{FP}\,p && \text{(add } c_{FP}\,p \text{ to both sides)} \\ c_{FP} &\le p\,\big(c_{FN} + c_{FP}\big) && \text{(factor out } p \text{)} \\ p &\ge \frac{c_{FP}}{c_{FP} + c_{FN}} && \text{(divide by the positive constant } c_{FP} + c_{FN} \text{)}, \end{aligned}$$

where $p$ is the score, i.e. the conditional probability that the example is positive. Expected total cost is the sum of conditional per-example costs, so minimizing each term yields a minimizing rule. This argument does not require the observed labels to be independent. At equality either action has the same conditional expected cost; our convention predicts positive. Writing $p^{*}=c_{FP}/(c_{FP}+c_{FN})$ for the boundary probability, its posterior odds equal the cost ratio,

$$\frac{p^{*}}{1 - p^{*}} = \frac{c_{FP}}{c_{FN}}.$$

With $c_{FP} = 5$ and $c_{FN} = 150$ the threshold is $5/155 \approx 0.032$, far below the default 0.5, and the required odds are 1 to 30. Equal error costs give the cutoff 0.5 in this model. Unequal costs give a different algebraic cutoff, although a population with no scores between the two cutoffs can yield identical decisions. Class imbalance alone does not determine the error costs.

The closed form has a precondition that is easy to miss. It requires the stated calibrated-probability assumption for the relevant population; merely lying between 0 and 1 is insufficient. When that assumption is unavailable, we drop the closed form and minimize the empirical cost $\hat{C}(t)$ over the threshold grid directly. This evaluates the available deterministic threshold states without assuming a probability scale. Its validation minimizer is an empirical choice, not a proof that it equals the population Bayes rule. What we lose is the ability to check the answer against a formula, and what we gain is freedom from the calibration assumption.

The selection protocol follows from the fact that $\hat{C}(t)$ is estimated. We need three disjoint roles. The training split fits the model that produces the scores for you to threshold. The validation split selects the threshold, by minimizing $\hat{C}(t)$ over its grid. The test split is touched once, to report the cost of the chosen rule. Selection can make the reported validation minimum optimistic in repeated samples; using test data to choose the threshold contaminates the only honest measurement we have. This is the same discipline as cross-validation, applied to a parameter that happens to be a scalar, and it is the reason hyperparameter search and nested resampling are natural companions to what we build here.

• Monotonicity. For $t_1 \le t_2$, every prediction made at $t_2$ is also made at $t_1$, so $TP(t_1) \ge TP(t_2)$ and $FP(t_1) \ge FP(t_2)$. Recall and the false positive rate are therefore nonincreasing in $t$, and the ROC curve is traced monotonically from $(0,0)$ to $(1,1)$.
• Rank invariance. If $g$ is strictly increasing, then $\mathbf{1}\{g(s_i) \ge g(t)\} = \mathbf{1}\{s_i \ge t\}$, so the entire threshold sweep, and hence the ROC curve, AUC, and the PR curve, are unchanged by replacing $s$ with $g(s)$.
• AUC as a probability. With continuous scores, $\text{AUC} = \Pr(s^{+} > s^{-})$ for independent draws from the positive and negative classes. The empirical estimate is the Mann-Whitney $U$ statistic divided by $N_0 N_1$, which is why AUC and the Wilcoxon rank-sum test are the same computation in different clothing.
• Prevalence effects. Under a pure prevalence shift with the class-conditional score distributions fixed, population ROC coordinates and AUC are unchanged. Precision and average precision can change; this need not happen under perfect separation. More general population changes can alter ROC behavior too.
• $F_1$ bounds. For defined nonnegative precision and recall with $P+R>0$, write $a = \min(P, R)$ and $b = \max(P, R)$. Then the harmonic mean satisfies $a \le 2ab/(a+b) \le \sqrt{ab} \le (a+b)/2$. The lower bound follows from $2b/(a+b) \ge 1$, so $F_1$ never falls below the smaller of the two inputs, and it never exceeds their arithmetic mean.
• Cost plateaus. Under the inclusive rule, consecutive observed scores $a<b$ give identical predictions for every $t\in(a,b]$. Empirical cost is constant on such a plateau. Distinct plateaus may tie for minimum cost, so the set of minimizing thresholds can be a union of intervals. Reporting a representative threshold does not imply its rounded value remains in the same plateau.
• Undefined corners. Precision is $0/0$ when no example is predicted positive, and recall is $0/0$ when the sample contains no positives at all. Both cases need an explicit convention, and the two curves in software differ in whether the corner point is returned.