Loss Functions › Visualizing Loss Behavior

Michael BrenndoerferMay 2, 20256 min read
Language AI HandbookAll chapters
Part 10 of 67Neural Network Foundations
Chapter 64Loss Functions › Visualizing Loss Behavior
Reading section 14 of 17 · Visualizing Loss Behavior · 6 minSection 14 of 17 · 6 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 64 of 513

Section 14 of 17 · 6 min read

Loss Functions › Visualizing Loss BehaviorOne section · Sections in this chapter14 / 17 ↓
Sections in this chapter17
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Out[20]:
Visualization
Line plot showing three regression loss functions versus prediction error from -4 to 4.
Comparison of MSE, MAE, and Huber loss as a function of the prediction error. MSE grows quadratically and penalizes large errors much more heavily than small ones. MAE grows linearly and is robust to outliers. Huber loss uses quadratic growth for small errors (below delta=1, marked by dotted vertical lines) and linear growth for large errors, combining the benefits of both.
Out[21]:
Visualization
Line plot showing cross-entropy loss versus predicted probability for true class, for standard CE and two label-smoothed variants.
Cross-entropy loss as a function of the predicted probability for the true class. The loss approaches infinity as the predicted probability approaches zero, creating a strong penalty for confident wrong predictions. Standard cross-entropy (solid line) is compared with label-smoothed versions using epsilon=0.05 and epsilon=0.1, which prevent infinite loss by requiring the model to assign small probability mass to all classes.
Out[22]:
Visualization
Line plot of focal loss versus predicted probability for gamma values 0, 1, 2, and 5.
Focal loss with different values of the focusing parameter gamma, compared against standard cross-entropy (gamma=0). Higher gamma values suppress the loss for well-classified examples more aggressively, forcing the model to focus on hard, uncertain examples where the model is currently wrong.
Line plot of the modulating factor (1-p)^gamma versus predicted probability for different gamma values.
The modulating factor (1-p)^gamma that focal loss multiplies against cross-entropy. At gamma=2, an example the model predicts with 90% confidence has its loss reduced by 100x compared to an uncertain example at 50% confidence.
Out[23]:
Visualization
Grouped bar chart showing softmax probabilities for three temperature values across four classes.
Effect of temperature scaling on softmax probability distributions applied to the same logit vector [3.0, 1.0, 0.5, 0.2]. Low temperature (T=0.5) sharpens the distribution, concentrating nearly all probability on Class 0. High temperature (T=5) flattens the distribution toward uniform. Temperature T=1 gives the standard softmax output.
Out[24]:
Visualization
Histogram comparing predicted probabilities for the true class under standard CE versus label smoothing.
Distribution of predicted probabilities for the true class under standard cross-entropy and label smoothing (epsilon=0.1), simulated from trained model logit distributions. Standard CE encourages extreme confidence, pushing many predictions toward 1.0. Label smoothing prevents this, producing a more evenly distributed confidence profile.
Bar chart comparing average logit gap under standard CE versus label smoothing.
Comparison of the average logit gap (difference between top logit and second-highest logit) under standard CE and label smoothing. Label smoothing significantly reduces the logit gap, which leads to better-calibrated representations and reduced overconfidence in deployment.