Beam Search › Length Normalization

Raw log-probability scores are biased against longer sequences. Since every additional token adds a negative log-probability term, longer sequences accumulate more negative contributions and systematically score lower than shorter ones. A model using raw scores will prefer short outputs, sometimes generating incomplete or truncated translations.

To see why this matters, imagine two candidate translations of a German sentence: a five-word English translation and a ten-word English translation. If each token has an average log-probability of -0.7 (a reasonable value for a good translation), the five-word translation scores -3.5 and the ten-word translation scores -7.0. The model will always prefer the five-word translation, even if the ten-word one is linguistically better. The model effectively penalizes every word added to the output.

The length penalty addresses this by normalizing the score by sequence length. One common formulation, introduced by Wu et al. in Google's Neural Machine Translation system, uses:

$$\text{score}(\mathbf{y}) = \frac{\log P(\mathbf{y})}{\text{lp}(\mathbf{y})}$$

where the length penalty $\text{lp}(\mathbf{y})$ is:

$$\text{lp}(\mathbf{y}) = \frac{(5 + |\mathbf{y}|)^{\alpha}}{(5 + 1)^{\alpha}}$$

where:

• $|\mathbf{y}|$: the length of the generated sequence (number of tokens)
• $\alpha$: a hyperparameter controlling how strongly length is penalized, typically set to 0.6 or 0.7
• The constant 5 is a smoothing term that prevents the penalty from being too aggressive for very short sequences

When $\alpha = 0$, the penalty is 1.0 for all lengths, and there is no length normalization (equivalent to raw log-probability). When $\alpha = 1$, the penalty is approximately linear in length, giving nearly equal footing to sequences of different lengths. Values in between, such as 0.6, provide a moderate preference for longer outputs without going all the way to strict length normalization.

Why not simply divide by $|\mathbf{y}|$ directly? The constant 5 in the numerator and denominator prevents very short sequences from being penalized too heavily and provides a smoother transition. Without it, the difference between length 1 and length 2 would be enormous (a factor of 2), while the difference between length 50 and length 51 would be negligible.

The choice of $\alpha$ has practical consequences. A value of $\alpha = 0.6$ was calibrated empirically on production MT systems. For summarization, where you generally want outputs shorter than the source, a lower $\alpha$ (or even $\alpha = 0$) may be more appropriate. For caption generation, where captions should have a natural length, $\alpha$ around 0.5 to 0.8 tends to work well. Tuning $\alpha$ on a validation set is standard practice.