Sinusoidal Position Encoding › Extrapolation Beyond Training Length
Language AI HandbookAll chapters Sinusoidal Position Encoding › Extrapolation Beyond Training LengthOne section · Sections in this chapter8 / 15 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
A significant advantage of sinusoidal encodings is their ability to represent positions never seen during training. Unlike learned position embeddings that require a fixed vocabulary of positions, sinusoidal encodings are computed from a deterministic formula that works for any position value.
This matters in practice. During training, you choose a maximum sequence length and process all examples within that budget. But at inference time, users sometimes send inputs longer than anything in the training set. With learned embeddings, position 513 in a model trained on sequences of length 512 simply does not exist; the embedding table has no entry for it, and you must either truncate the input or handle a missing value. With sinusoidal encoding, position 513 is just another application of the formula, and the output is a perfectly valid encoding vector.
Let's examine how encodings behave beyond typical training lengths:
# Generate encodings for much longer sequences
extended_max_len = 10000
PE_extended = sinusoidal_position_encoding(extended_max_len, d_model)
# Check that encodings remain bounded
pos_samples = [0, 100, 1000, 5000, 9999]
encoding_norms = [np.linalg.norm(PE_extended[p]) for p in pos_samples]Encoding statistics for extended positions: -------------------------------------------------- Position 0: L2 norm = 5.6569 Position 100: L2 norm = 5.6569 Position 1000: L2 norm = 5.6569 Position 5000: L2 norm = 5.6569 Position 9999: L2 norm = 5.6569 All values remain in [-1, 1] by construction. L2 norms are similar because encodings use orthogonal sine/cosine pairs.
The encodings remain well-behaved even at position 9,999. Each dimension independently oscillates between -1 and 1, so the encoding never explodes or vanishes. The L2 norms are all similar because the encoding uses independent unit-norm dimension pairs, each contributing approximately equally to the total norm.
However, extrapolation has a subtle limitation. While the encodings themselves are mathematically valid for any position, the model's attention patterns are learned on sequences of a particular length distribution. If the model trains on sequences of length 512, it has never seen the specific encoding patterns that occur at position 5000. The attention mechanism might not generalize well to these unseen patterns, even though the encodings are perfectly valid. This gap between mathematical validity and practical performance is one of the important nuances of sequence-length generalization, and it motivates much of the later work on position encoding in long-context models.
