Uncertainty Quantification › Visualizations

Michael BrenndoerferFebruary 24, 20265 min read
Language AI HandbookAll chapters
Part 56 of 67Hallucination and Factuality
Chapter 434Uncertainty Quantification › Visualizations
Reading section 7 of 11 · Visualizations · 5 minSection 7 of 11 · 5 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 434 of 513

Section 7 of 11 · 5 min read

Uncertainty Quantification › VisualizationsOne section · Sections in this chapter7 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's visualize the key concepts with figures that make the ideas concrete.

Out[16]:
Visualization
Reliability diagram comparing overconfident and temperature-scaled model calibration against the ideal diagonal.
Reliability diagrams comparing the overconfident model (coral) against the temperature-scaled model (blue) and perfect calibration (dashed diagonal). The overconfident model's curve lies below the diagonal, meaning it claims higher confidence than its accuracy justifies. After temperature scaling, the calibrated model tracks the diagonal much more closely, especially in the high-confidence region where most predictions fall.
Out[17]:
Visualization
Line plot of agreement and entropy vs question difficulty, with agreement decreasing and entropy increasing toward the hardest questions.
Agreement score and binary entropy as a function of question difficulty. High-difficulty questions show lower agreement and higher entropy, confirming that sampling-based methods correctly identify uncertain questions. Entropy peaks for the hardest questions, where sampled answers approach an even split.
Noisy line plot of semantic entropy vs question difficulty, showing a clear upward trend.
Semantic entropy generally increases with question difficulty despite sampling noise. Unlike token-level entropy, semantic entropy is invariant to surface-form variation and directly captures the spread of distinct meaning clusters across sampled responses, making it a cleaner uncertainty signal.
Out[18]:
Visualization
Bar chart comparing ECE and calibration gap for uncalibrated, temperature-scaled, and Platt-scaled models.
ECE and calibration gap across three methods: uncalibrated, temperature-scaled, and Platt-scaled. Both calibration methods substantially reduce ECE compared to the raw overconfident baseline, with the calibration gap (mean confidence minus mean accuracy) also dropping toward zero. Temperature scaling uses one parameter; Platt scaling uses two and can correct asymmetric miscalibration.