Classification Metrics and Decision Thresholds

A risk model produces a score; an alerting rule decides which scores lead to action; you choose where that cut sits. Suppose a transaction has a true conditional fraud probability of 0.04, a false alarm costs $5 and a missed fraud costs $150. With zero cost for correct decisions, alerting has expected cost 0.96 times $5, or $4.80, while letting it pass has expected cost 0.04 times $150, or $6. Under these assumptions, alerting costs less. An arbitrary score of 0.04 does not establish that probability: both the score scale and the consequences of the decision matter.

Class imbalance makes accuracy alone a poor guide to a decision. Suppose one transaction in fifty is fraudulent. A model that flags nothing is right 98 percent of the time but misses every fraud. A model that flags everything catches every fraud and sends every transaction for review. Accuracy happily reports 98 percent for the first and 2 percent for the second, so we need measurements that separate the two kinds of mistakes instead of merging them into one number.

We fix notation once and use it throughout. We start with the confusion matrix, the four counts every binary decision produces, and read precision, recall, and the false positive rate directly off it. We then follow those quantities as the threshold slides, which produces the threshold curves, the ROC curve, and the precision-recall curve, along with their summary numbers, ROC AUC and average precision. Finally we attach costs to the two mistakes and show that the threshold is not a matter of taste but an optimization problem: minimize observed cost on validation data, then evaluate the fixed rule once on test data.

Two distinctions run through the whole chapter and are worth stating early for you. The first is between ranking and calibration. ROC AUC and average precision care only about the order in which a model sorts examples; they are unchanged if we replace every score by its cube. The analytic probability cutoff needs a meaningful probability scale; empirical cost selection can instead evaluate arbitrary scores directly. The second is between measurement and decision. Metrics summarize how a fixed classifier behaves; a threshold turns a score into a classifier. Choosing the threshold is model selection in miniature, and the same leakage rules from Validation Strategies and Data Leakage apply: the data used to pick the threshold cannot also provide an independent evaluation of that choice.