Probability Calibration › Visualizing Probability Calibration

The reliability diagram is the main diagnostic for calibration. We compute it on the untouched test set. The deliberately sharpened raw scores illustrate scale miscalibration. The sigmoid map estimates a smooth correction, but its curve on the separate test partition need not lie on the diagonal. Isotonic recalibration can bend more sharply, but it can also create flat segments where many scores receive the same calibrated probability.

The reliability diagram shows how closely predicted probabilities track observed rates across the probability scale.

The histogram shows where the probability mass sits. An overconfident classifier often places many test cases near zero or one. Calibration moves some of that mass toward the middle, but it does not necessarily make the distribution symmetric. The shape of the calibrated distribution depends on the fitted map and on the raw scores. We should not interpret a concentration near the base rate as a failure by itself; the relevant question is whether the observed rates in each region match the predicted probabilities.

The score comparison makes the distinction between calibration and discrimination concrete. The Brier score and log loss are proper scores that reward both calibration and separation. AUC is a ranking metric. A strictly increasing map preserves AUC before numerical clipping or saturation; nondecreasing isotonic maps can change ties. Proper scores may improve or worsen on new cases. In the chart, lower bars indicate better probability quality. Compare the methods on each score, then use the separate AUC table to check ranking. An estimated recalibrator can worsen a test score, and isotonic plateaus can change AUC by creating ties. If a method appears to improve AUC substantially, we should check for leakage, a change in the test set, or a nonmonotone map that is doing something other than calibration.