Layer Normalization › Epsilon: A Small but Critical Detail

Michael BrenndoerferUpdated June 11, 20253 min read
Language AI HandbookAll chapters
Part 15 of 67Transformer Blocks
Chapter 104Layer Normalization › Epsilon: A Small but Critical Detail
Reading section 10 of 16 · Epsilon: A Small but Critical Detail · 3 minSection 10 of 16 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 104 of 513

Section 10 of 16 · 3 min read

Layer Normalization › Epsilon: A Small but Critical DetailOne section · Sections in this chapter10 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The epsilon parameter (ϵ\epsilon) appears in the denominator of the normalization formula:

x^i=xi−μσ2+ϵ\hat{x}_i = \frac{x_i - \mu}{\sqrt{\sigma^2 + \epsilon}}

where:

  • x^i\hat{x}_i: the normalized value for the ii-th feature
  • xix_i: the original input value
  • μ\mu: the mean across all features
  • σ2\sigma^2: the variance across all features
  • ϵ\epsilon: a small constant added to prevent division by zero

The purpose of ϵ\epsilon is to ensure numerical stability. If all input values are identical (or nearly so), the variance σ2\sigma^2 approaches zero. Without ϵ\epsilon, we would divide by zero, producing infinity or NaN. Adding a small positive constant like 10−510^{-5} ensures the denominator is always positive.

The choice of epsilon can affect numerical stability, especially with mixed-precision (FP16) training where very small values may underflow.

In mixed-precision training, activations are stored as 16-bit floating-point numbers, which have a minimum representable positive value of approximately 6×10−86 \times 10^{-8}. If the variance of a representation is smaller than this threshold, the FP16 representation will round it to zero, causing the same division-by-zero problem that epsilon was designed to prevent. For FP16 training, it is common to use a larger epsilon (around 10−410^{-4} or 10−310^{-3}) to ensure stability even when the FP16 representation truncates small variance values. This concern is practical: several practitioners have traced training instabilities in large language models to epsilon values that were appropriate for FP32 but insufficient for FP16.

In[25]:
Code
# Demonstrate epsilon's role with near-constant input
near_constant = torch.ones(1, 4, 8) * 5.0
near_constant[0, 0, 0] = 5.001  # Tiny variation


def test_epsilon(x, eps):
    """Test layer normalization with different epsilon values."""
    mu = x.mean(dim=-1, keepdim=True)
    var = x.var(dim=-1, keepdim=True, unbiased=False)
    try:
        x_norm = (x - mu) / torch.sqrt(var + eps)
        return x_norm.std().item(), "OK"
    except Exception as e:
        return float("nan"), str(e)


epsilons = [0, 1e-12, 1e-8, 1e-5, 1e-3]
Out[26]:
Console
Effect of epsilon on near-constant input:
  Input variance: 1.25e-07

Epsilon      Output std      Status
----------------------------------------
0e+00        NaN/Inf         Division issue
1e-12        0.507998        OK
1e-08        0.486255        OK
1e-05        0.052836        OK
1e-03        0.005312        OK

The input variance is extremely small (around 10−810^{-8}), which means we're dividing by a very small number. With epsilon = 0, the output standard deviation explodes because we're essentially dividing by nearly zero. As epsilon increases, the output becomes more stable. The standard choice of 10−510^{-5} strikes a balance: it's large enough to prevent numerical issues but small enough not to distort the normalization when variance is reasonably sized. A reasonable epsilon value (typically 10−510^{-5} to 10−610^{-6}) provides a safety net without affecting normal computations.