Classification Metrics and Decision Thresholds › Disadvantages

Every one of these metrics is a sample estimate, and at low prevalence the estimates are noisy precisely where it counts. Precision has predicted positives as its denominator, so few alerts can make its estimate sensitive to sampling. AUC uncertainty depends on both class counts and the score distributions. A point estimate describes the observed result but does not describe that uncertainty.

ROC AUC's indifference to prevalence is a feature for ranking comparisons and a trap for decision making. Because the false positive rate divides by the number of negatives, a model can achieve a flattering AUC while producing an unusable number of false alerts when negatives vastly outnumber positives. A high AUC also says nothing about the operating point we intend to use: two models with identical AUC can behave very differently in the low false positive rate region where a real review team lives.

Average precision fixes the prevalence problem in one direction and creates it in another. It depends on the class balance of the evaluation set, so an average precision of 0.6 measured on a sample with 5 percent positives cannot be assumed to hold at 1 percent prevalence. Re-evaluate or adjust under justified transport assumptions; a change is not inevitable, as perfect separation illustrates.

Calibration is a separate property that neither AUC nor average precision measures. Both are ranking statistics, blind to any strictly increasing rescaling of the scores. Empirical cost minimization can use a rank-only score, whereas the closed-form rule $c_{FP}/(c_{FP}+c_{FN})$ requires the stated probability assumption. A value between 0 and 1, including a vote fraction, is not by itself evidence of calibration for the deployment population.

Finally, threshold selection is itself a fitting procedure, so it consumes data. For a fixed candidate family whose individual validation cost estimates are unbiased, the expected validation minimum is no greater than the smallest population cost in that family. This expectation does not dictate the ordering in one realized validation/test comparison. The size of selection optimism depends on the joint variability of the estimates and the selection procedure; a flat cost curve alone does not bound it. Independent held-out observations evaluate the selected fixed rule.