Classification Metrics and Decision Thresholds › Advantages

Metrics built on the confusion matrix tell us things accuracy hides. Precision answers the question the review team actually asks, which is what fraction of the alerts you are about to work through are real. Recall answers the question you ask as a risk owner, which is what fraction of the bad events we are catching. Keeping these separate makes the tradeoff visible instead of averaging it away, and it makes the tradeoff negotiable: we can move along the curve deliberately rather than discovering the operating point by accident.

The threshold is an inexpensive and effective control you can turn. Once a model is trained, moving the threshold changes the operating point without retraining, and the whole family of achievable confusion matrices can be read off in a single sweep of order $n \log n$ operations. For a fixed score and the inclusive threshold convention, the sweep enumerates every achievable deterministic threshold state, including the endpoints and whole tied groups.

Curve summaries are useful for comparing models before you fix any threshold. ROC AUC has a probabilistic meaning: the probability that an independent positive outranks an independent negative, with half credit for ties. It is invariant to strictly increasing transformations of the score, so a model is not penalized for being badly calibrated when the question is only about ordering. Average precision focuses the same idea on the positive class and is more informative when positives are rare, because it is sensitive to the false positives that actually compete with true positives for review capacity.

Cost-based threshold selection converts an abstract tradeoff into a decision. Naming $c_{FP}$ and $c_{FN}$ forces the organization to state what each kind of mistake costs, and the population objective identifies a threshold that minimizes expected cost, while its empirical estimate selects a threshold from validation observations. When scores are calibrated, the optimal threshold has a closed form, $c_{FP}/(c_{FP}+c_{FN})$, which is easy to audit and easy to challenge in a meeting.