Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

RMSNorm › Parameter Efficiency

Michael BrenndoerferUpdated June 7, 20253 min read

Language AI HandbookAll chapters

Part 15 of 67Transformer Blocks

Chapter 105RMSNorm › Parameter Efficiency

Reading section 7 of 15 · Parameter Efficiency · 3 minSection 7 of 15 · 3 min

Section 7 of 15 · 3 min read

RMSNorm › Parameter EfficiencyOne section · Sections in this chapter7 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Beyond computational cost, RMSNorm also reduces the number of learnable parameters. LayerNorm has two learnable vectors per layer ( $\gamma$ and $\beta$ ), while RMSNorm has only one ( $\gamma$ ). This halves the number of normalization parameters throughout the model.

The $\beta$ parameter in LayerNorm is a learned bias: after normalizing to zero mean, $\beta$ shifts the output to any desired mean. In a pre-norm transformer architecture (which is the standard today), this learned offset is mostly redundant because the linear projections in the attention and feed-forward sub-layers have their own bias terms that can achieve the same effect. Removing $\beta$ from the normalization layer does not remove the model's ability to learn output offsets; it just removes one redundant mechanism for doing so.

The parameter savings are exact and predictable: every normalization layer saves exactly hidden_dim parameters. For a model with many normalization layers, this adds up.

In[16]:

Code

def count_norm_parameters(hidden_dim, num_layers, norm_type="rmsnorm"):
    """Count normalization parameters in a transformer."""
    params_per_layer = {
        "layernorm": 2 * hidden_dim,  # gamma and beta
        "rmsnorm": hidden_dim,  # gamma only
    }
    return params_per_layer[norm_type] * num_layers


# Compare for different model sizes
model_configs = [
    ("GPT-2 Small", 768, 12),
    ("GPT-2 Medium", 1024, 24),
    ("GPT-2 Large", 1280, 36),
    ("LLaMA 7B", 4096, 32),
    ("LLaMA 13B", 5120, 40),
]

Out[17]:

Visualization

Horizontal bar chart comparing LayerNorm and RMSNorm parameter counts across model sizes from GPT-2 Small to LLaMA 13B. — Normalization parameter counts for different model sizes, comparing LayerNorm (two vectors per layer) with RMSNorm (one vector per layer). The parameter savings scale linearly with model size: LLaMA 7B saves over 500K normalization parameters, each of which also requires gradient computation and optimizer state storage during training.

The parameter savings scale with model size. For LLaMA 7B, switching from LayerNorm to RMSNorm saves over 500,000 parameters. While this is less than 0.01% of the total model size, these parameters also consume memory bandwidth during inference and require gradient computation during training. In the context of training with Adam optimizer, each parameter also requires two additional optimizer state tensors (first and second moment estimates), so the actual memory saving is roughly three times the raw parameter count. Every reduction helps, and at the scale of modern LLMs, even fractions of a percent matter.