RMSNorm › Mathematical Connection Between RMS and Standard Deviation

We've claimed that RMSNorm works because activations in neural networks tend to be near-centered. But how near is near enough? To answer this precisely, we need to understand the mathematical relationship between what RMSNorm computes (the RMS) and what LayerNorm computes (the standard deviation). If the two quantities are nearly identical under typical transformer conditions, then the two normalizations are nearly identical in their effect.

This section derives the exact connection between these two quantities. The derivation is worth following carefully because it reveals a beautiful geometric relationship and tells us exactly when the two normalizations diverge. Understanding this relationship gives you the theoretical confidence to choose RMSNorm over LayerNorm: you will know precisely the conditions under which the choice matters and when it does not.

The key insight is that RMS and standard deviation are related through a Pythagorean identity involving the mean. This identity makes the trade-off perfectly precise: the larger the mean of the activations, the more the two normalizations differ. Conversely, when the mean is small relative to the standard deviation, the two normalizations are nearly interchangeable.

We want to express $\text{RMS}(\mathbf{x})$ in terms of $\sigma$ (the standard deviation) and $\mu$ (the mean). If we can do this, we'll know how much the two normalizations differ based on the mean alone.

Start with the definition of variance, which measures how spread out values are around their mean. The variance is defined as the average squared deviation from the mean:

$$\sigma^2 = \frac{1}{d} \sum_{i=1}^{d} (x_i - \mu)^2$$

where:

• $\sigma^2$: the variance of the input vector, measuring spread around the mean
• $d$: the number of elements in the vector
• $x_i$: the $i$-th element of the input vector
• $\mu$: the mean of the input elements, equal to $\frac{1}{d}\sum_{i=1}^{d} x_i$

This formula takes each element, measures its distance from the mean, squares that distance, and averages all the squared distances. The square root of variance gives us the standard deviation $\sigma$, which has the same units as the original data.

Our strategy is to expand the variance formula and recognize familiar terms. We'll use the algebraic identity $(a - b)^2 = a^2 - 2ab + b^2$ to expand the squared term:

$$\sigma^2 = \frac{1}{d} \sum_{i=1}^{d} (x_i^2 - 2x_i\mu + \mu^2)$$

Now we can distribute the sum across the three terms. Each term gets its own summation:

$$\sigma^2 = \frac{1}{d} \sum_{i=1}^{d} x_i^2 - \frac{2\mu}{d} \sum_{i=1}^{d} x_i + \frac{1}{d} \sum_{i=1}^{d} \mu^2$$

Let's simplify each term:

• First term: $\frac{1}{d} \sum_{i=1}^{d} x_i^2$ is the mean of squared values. This is exactly $\text{RMS}(\mathbf{x})^2$.
• Second term: The sum $\sum_{i=1}^{d} x_i$ equals $d \cdot \mu$ by the definition of mean. So this term becomes $\frac{2\mu}{d} \cdot (d\mu) = 2\mu^2$.
• Third term: We're summing the constant $\mu^2$ exactly $d$ times, so this equals $\frac{d \cdot \mu^2}{d} = \mu^2$.

Substituting these simplifications:

$$\sigma^2 = \text{RMS}(\mathbf{x})^2 - 2\mu^2 + \mu^2 = \text{RMS}(\mathbf{x})^2 - \mu^2$$

Rearranging to isolate the RMS:

$$\text{RMS}(\mathbf{x})^2 = \sigma^2 + \mu^2$$

Taking square roots of both sides:

$$\text{RMS}(\mathbf{x}) = \sqrt{\sigma^2 + \mu^2}$$

where:

• $\text{RMS}(\mathbf{x})$: the root mean square of the input vector
• $\sigma$: the standard deviation of the input vector (spread around the mean)
• $\mu$: the mean of the input vector (location of the center)

Why does this formula make sense? Notice that the RMS combines information about both the mean and the spread of the distribution. The standard deviation $\sigma$ only captures spread: it measures how far values deviate from their center. The mean $\mu$ captures location: it measures where the center is. The RMS captures both because it measures distance from zero, not from the mean. A vector with large mean but small variance still has large RMS.

This formula has a beautiful geometric meaning. Think of $\sigma$ and $\mu$ as the two legs of a right triangle. The RMS is the hypotenuse. The Pythagorean theorem tells us that $\text{hypotenuse}^2 = \text{leg}_1^2 + \text{leg}_2^2$, which is exactly what we derived.

Think of it this way: if you draw a 2D coordinate system where the horizontal axis represents "standard deviation" and the vertical axis represents "mean," then any distribution maps to a point at coordinates $(\sigma, \mu)$. The RMS is the distance from the origin to that point, the length of the line segment from $(0,0)$ to $(\sigma, \mu)$. LayerNorm normalizes by the horizontal coordinate (divides by $\sigma$). RMSNorm normalizes by the diagonal distance from the origin (divides by $\sqrt{\sigma^2 + \mu^2}$). When the vertical coordinate (the mean) is small, these two quantities are nearly identical.