Probability Calibration

A classifier that returns predict_proba gives us estimated probabilities, but those estimates are not automatically calibrated. A model may say 0.90 for a group of cases in which only 70 percent are positive. The ranking may still be excellent, and the classifier may still be useful for sorting cases, but the number 0.90 cannot be read as a 90 percent chance. Probability calibration is the problem of making predicted probabilities match observed frequencies.

Many decisions use the probability itself. A fraud team may set a review threshold from expected costs, a hospital may use a risk score to allocate follow-up care, and a lending model may be monitored by comparing predicted default rates with realized default rates. In each case, a score of 0.2 should mean roughly a 20 percent event rate, not merely “lower than a score of 0.3.” Calibration is also distinct from discrimination. A model can rank positives above negatives very well while being badly overconfident, and a model can be perfectly calibrated while having no ability to separate the two classes.

Calibration is a property of the model and the population together, not a property of the model alone. A calibrator learned on one population can fail on another. We therefore treat calibration as a monitored, evidence-based step: fit it on representative held-out data, evaluate it on untouched data, and check whether the reliability holds across important subgroups and over time.