Learned Position Embeddings › Summary

Learned position embeddings treat position representations as trainable parameters rather than fixed formulas. This simple idea has proven remarkably effective across many language modeling tasks and underpins some of the most influential transformer architectures ever built.

Key takeaways from this chapter:

• Position embedding table: A learnable matrix of shape $(L_{\max}, d)$ stores one embedding vector per position. Position $i$ is represented by the $i$-th row of this table, analogous to how token embeddings work.

• Additive combination: Position embeddings are added to token embeddings at the input layer, creating representations that encode both what token appears and where it appears. The sum is passed to all subsequent attention and feed-forward layers.

• Training discovers structure: Through backpropagation, position embeddings learn to encode positional information useful for the task. Nearby positions typically develop similar embeddings, and the embeddings form a smooth low-rank manifold in the embedding space.

• Maximum sequence length constraint: Unlike sinusoidal encodings, learned embeddings cannot represent positions beyond $L_{\max}$. Processing longer sequences requires techniques like position interpolation (which stretches the existing embedding table) or fine-tuning with extended context.

• Low effective dimensionality: Despite being stored in high-dimensional space, position embeddings often lie in a low-rank subspace. A few principal components capture most of the positional information, suggesting that the fundamental structure of position is simple even when represented in high dimensions.

• GPT and BERT style: Major models like GPT-2 and BERT use learned position embeddings with simple additive combination. BERT adds segment embeddings on top to support sentence-pair inputs. The approach is straightforward to implement and works well in practice.

• Trade-offs vs. sinusoidal: Learned embeddings offer more flexibility and task-adaptability but are limited in length generalization. Sinusoidal encodings generalize to any length but impose a fixed mathematical structure. For most large-scale language modeling applications, learned embeddings are the practical default.

• PCA geometry: The PCA projection of trained position embeddings reveals an ordered curve through the low-dimensional subspace, confirming that training imposes a smooth, ordered geometry consistent with the sequence structure of language.

In the next chapter, we'll explore relative position encodings, which address the extrapolation problem by encoding the distance between positions rather than their absolute locations. This design helps models generalize to different sequence lengths without interpolation tricks.