Probability Calibration › Summary

Probability calibration makes predicted probabilities match observed frequencies. It is separate from discrimination: a model can rank cases well while being overconfident, and a model can be calibrated while having no ranking power. Reliability diagrams inspect calibration, while Brier score and log loss evaluate overall probability quality. Brier score and log loss are proper scoring rules, meaning they are minimized in expectation by the true probability. AUC measures ranking and is not a calibration metric. The expected calibration error is a useful diagnostic, but it depends on binning and is not proper.

Sigmoid recalibration fits a two-parameter logistic map on the logit scale; its limited flexibility can help with smaller samples, but estimation still needs adequate class and score support. Isotonic recalibration uses the pool adjacent violators algorithm for monotone block estimates and interpolates between score thresholds for prediction. Both methods are post hoc, need separate calibration data, and consume only the existing model score. Strictly increasing recalibration preserves ranking. Ties and negative fitted slopes need separate checks; calibration should be evaluated for probability reliability rather than promoted as a ranking improvement.

The practical workflow is to train the base model, calibrate on a separate representative partition, and evaluate on untouched test data with reliability diagrams, Brier score, log loss, AUC, and subgroup checks. Calibration is a property of the model-data pair, not a permanent guarantee. It should be monitored over time and reassessed after population or base-rate changes. Re-estimate the probability map when evidence from the target population supports an update, and revalidate the decision rule when costs or decision context change.