Classification Metrics and Decision Thresholds › Summary

A score is not a decision. Turning a score into a decision requires a threshold, and the threshold is chosen by trading false positives against false negatives. The confusion matrix records that trade at one point; precision, recall, and the false positive rate read the trade off the matrix; the threshold curves, ROC curve, and precision-recall curve show the whole family of trades available from a fixed ranking.

The summary numbers answer narrow questions. ROC AUC compares independent positive and negative draws, awarding half credit for ties. It is invariant to strictly increasing score transformations and is unchanged by a prevalence-only shift that preserves both class-conditional score distributions. Average precision concentrates on the positive class, summarizes the precision-recall curve using the stated noninterpolated convention, and can depend on prevalence, so it cannot be assumed to transfer unchanged between populations. Neither number identifies an operating point, and neither establishes calibration. The trapezoidal area under a precision-recall curve uses a different summation convention from average precision; state which one is reported.

The threshold itself is an optimization problem once costs are named. Minimizing expected cost gives the population objective $c_{FP}(1-\pi)\text{FPR} + c_{FN}\pi(1-\text{TPR})$, whose level sets are straight lines in ROC space with slope $c_{FP}(1-\pi)/(c_{FN}\pi)$. When the score is a calibrated probability, the optimum collapses to the closed form $c_{FP}/(c_{FP}+c_{FN})$, or equivalently to posterior odds equal to the cost ratio. When that probability assumption is unavailable, sweeping validation cost still selects an empirical threshold for the given ranking; it does not certify the same population optimum.

The discipline that makes the resulting number trustworthy is the split discipline. Threshold selection is a fitting step, so it consumes validation data; the test split is touched once, to report. The selected threshold is typically not unique, since the cost is a step function with plateaus, and thresholds can vary across resamples while their observed costs remain similar. Threshold and cost dispersion have different units and should be reported on their own scales; these observations alone do not guarantee future-risk stability. Reporting a range and a cost, rather than three decimal places, is the honest summary, and it is what a team needs in order to know whether the decision is stable enough to automate.