Calibration › Calibration Plots (Reliability Diagrams)

A calibration plot, or reliability diagram, visualizes the relationship between predicted confidence and actual accuracy. This graphical representation makes it easy to identify systematic patterns of overconfidence or underconfidence across the confidence spectrum. To construct one:

1. Divide the confidence range $[0, 1]$ into $M$ bins
2. For each bin, plot the mean confidence on the x-axis and the empirical accuracy on the y-axis
3. Draw a diagonal reference line representing perfect calibration

Points falling below the diagonal indicate overconfidence (the model is less accurate than its confidence suggests). Points above the diagonal indicate underconfidence. The gap between the calibration curve and the diagonal line, weighted by bin size, corresponds to the ECE.

Modern reliability diagrams often include histograms showing the distribution of predictions across confidence values, revealing whether the model tends toward extreme confidences (common in overconfident models) or expresses more moderate uncertainty. A histogram showing most mass near 1.0 suggests an overconfident model that rarely admits uncertainty, while a uniform distribution indicates the model uses the full range of confidence values.