Calibration › Code Implementation
Language AI HandbookAll chapters Calibration › Code ImplementationOne section · Sections in this chapter8 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We implement calibration metrics and visualization using Python. We will create synthetic classification data with known calibration properties, then apply calibration methods.
import numpy as np
import torch
# Set random seed for reproducibility
np.random.seed(42)
torch.manual_seed(42)
# Generate synthetic data: 1000 samples, 5 classes
n_samples = 1000
n_classes = 5
# True labels
true_labels = np.random.randint(0, n_classes, n_samples)
# Simulate uncalibrated model outputs (overconfident)
# The model tends to be correct but with inflated confidence
logits = np.random.randn(n_samples, n_classes) * 2.0
# Boost the true class logit to simulate overconfidence
for i in range(n_samples):
logits[i, true_labels[i]] += 3.0
# Convert to probabilities
probs = torch.softmax(torch.tensor(logits), dim=1).numpy()
pred_labels = np.argmax(probs, axis=1)
confidences = np.max(probs, axis=1)We have created synthetic logits where the correct class receives an artificial boost, simulating the overconfidence common in neural networks. We examine the accuracy and confidence distribution.
Accuracy: 0.678 Mean Confidence: 0.740 Gap (Confidence - Accuracy): 0.062
The model shows higher average confidence than accuracy, indicating overconfidence. This gap between mean confidence and accuracy is a quick first diagnostic: a positive gap suggests overconfidence, and a negative gap suggests underconfidence. Now let us implement the Expected Calibration Error calculation.
def expected_calibration_error(y_true, y_prob, y_pred, n_bins=10):
"""
Calculate Expected Calibration Error.
Args:
y_true: True labels (integer indices)
y_prob: Predicted probabilities for the predicted class
y_pred: Predicted labels
n_bins: Number of bins for calibration
Returns:
ece: Expected Calibration Error
bin_accuracies: Accuracy per bin
bin_confidences: Confidence per bin
bin_counts: Number of samples per bin
"""
bin_boundaries = np.linspace(0, 1, n_bins + 1)
bin_lowers = bin_boundaries[:-1]
bin_uppers = bin_boundaries[1:]
ece = 0.0
bin_accuracies = []
bin_confidences = []
bin_counts = []
for bin_lower, bin_upper in zip(bin_lowers, bin_uppers):
# Find samples in this bin
in_bin = (y_prob > bin_lower) & (y_prob <= bin_upper)
prop_in_bin = np.mean(in_bin)
if prop_in_bin > 0:
accuracy_in_bin = np.mean((y_pred == y_true)[in_bin])
confidence_in_bin = np.mean(y_prob[in_bin])
ece += np.abs(accuracy_in_bin - confidence_in_bin) * prop_in_bin
bin_accuracies.append(accuracy_in_bin)
bin_confidences.append(confidence_in_bin)
bin_counts.append(np.sum(in_bin))
else:
bin_accuracies.append(0.0)
bin_confidences.append(0.0)
bin_counts.append(0)
return (
ece,
np.array(bin_accuracies),
np.array(bin_confidences),
np.array(bin_counts),
)
# Calculate ECE for our synthetic model
ece, bin_acc, bin_conf, bin_counts = expected_calibration_error(
true_labels, confidences, pred_labels, n_bins=10
)Expected Calibration Error: 0.0645 (6.45%) Bin statistics (10 bins): Bin Range Count Accuracy Confidence Gap ------------------------------------------------------------ [0.0, 0.1)0 0.000 0.000 +0.000 [0.1, 0.2)0 0.000 0.000 +0.000 [0.2, 0.3)2 1.000 0.264 -0.736 [0.3, 0.4)45 0.356 0.363 +0.008 [0.4, 0.5)100 0.430 0.452 +0.022 [0.5, 0.6)129 0.465 0.552 +0.087 [0.6, 0.7)145 0.607 0.650 +0.043 [0.7, 0.8)124 0.685 0.748 +0.063 [0.8, 0.9)150 0.733 0.851 +0.118 [0.9, 1.0)305 0.898 0.956 +0.058
Looking at the per-bin statistics, we can read the calibration curve numerically before visualizing it. Bins where the gap is positive indicate overconfidence: the model claims more confidence than its accuracy warrants. Bins where the gap is negative indicate underconfidence. The pattern of gaps across bins reveals whether the miscalibration is uniform (suggesting temperature scaling is appropriate) or localized to specific confidence ranges (suggesting a non-parametric method would be better).
The ECE shows significant miscalibration, with several bins showing large gaps between confidence and accuracy. We visualize this with a reliability diagram.
The reliability diagram confirms overconfidence: the model's accuracy is consistently lower than its confidence. The histogram shows that most predictions have high confidence (above 0.7), which is typical for overconfident neural networks. Now we implement temperature scaling to calibrate these probabilities.
def temperature_scaling(logits, y_true, eps=1e-10):
"""
Find optimal temperature for calibration using negative log-likelihood.
"""
import numpy as np
import torch
import torch.nn.functional as F
from scipy.optimize import minimize_scalar
def nll_loss(T):
# Apply temperature scaling
scaled_logits = logits / T
probs = F.softmax(torch.tensor(scaled_logits), dim=1).numpy()
# Calculate negative log-likelihood
nll = 0.0
for i in range(len(y_true)):
nll -= np.log(probs[i, y_true[i]] + eps)
return nll / len(y_true)
# Optimize temperature
result = minimize_scalar(nll_loss, bounds=(0.1, 10.0), method="bounded")
optimal_T = result.x
# Apply optimal temperature
scaled_logits = logits / optimal_T
calibrated_probs = F.softmax(torch.tensor(scaled_logits), dim=1).numpy()
return optimal_T, calibrated_probs
# Apply temperature scaling
optimal_T, cal_probs = temperature_scaling(logits, true_labels)
cal_pred_labels = np.argmax(cal_probs, axis=1)
cal_confidences = np.max(cal_probs, axis=1)Optimal Temperature: 1.305 Original Accuracy: 0.678 Calibrated Accuracy: 0.678 Original Mean Confidence: 0.740 Calibrated Mean Confidence: 0.671
The optimal temperature is greater than 1, which softens the distribution and reduces confidence. Notice that accuracy remains unchanged (temperature scaling preserves rankings), but mean confidence has decreased to better match the actual accuracy. The optimal temperature value itself is informative: larger values indicate more severe overconfidence that required more aggressive softening.
We calculate the ECE after calibration and visualize the improvement.
# Calculate ECE after temperature scaling
ece_cal, bin_acc_cal, bin_conf_cal, _ = expected_calibration_error(
true_labels, cal_confidences, cal_pred_labels, n_bins=10
)ECE before calibration: 0.0645 ECE after temperature scaling: 0.0235 Improvement: 0.0410 (63.6% reduction)
Temperature scaling has significantly reduced the ECE by bringing the calibration curve closer to the diagonal. The confidence distribution has also shifted leftward, reducing the overconfidence. Now we implement isotonic regression for comparison.
# For isotonic regression, we need binary outcomes (correct/incorrect)
correct = (pred_labels == true_labels).astype(float)
# Fit isotonic regression
from sklearn.isotonic import IsotonicRegression
iso_reg = IsotonicRegression(out_of_bounds="clip")
iso_reg.fit(confidences, correct)
iso_confidences = iso_reg.predict(confidences)
# Clip to valid probability range
iso_confidences = np.clip(iso_confidences, 0, 1)ECE with Temperature Scaling: 0.0235 ECE with Isotonic Regression: 0.0000
Isotonic regression often achieves lower ECE than temperature scaling because it is non-parametric and can fit complex calibration curves. However, it requires more data and can overfit on small validation sets. It also does not necessarily preserve the ranking of predictions when applied to out-of-distribution data, unlike temperature scaling, which maintains the argmax ordering.