Position Problem › In Practice: What Modern Models Use

Comparing these options helps interpret the design choices in state-of-the-art models. Here is a brief survey of what modern systems use; later chapters examine these methods in detail.

BERT (2018) uses absolute learned positional embeddings with a maximum sequence length of 512. These are trained from random initialization along with the word embeddings. BERT's success demonstrated that learned absolute embeddings work well for most NLP tasks, though they cannot generalize beyond the 512-position limit.

GPT-2 and GPT-3 also use absolute learned positional embeddings, with GPT-2 supporting 1024 tokens and GPT-3 supporting 2048. These are pure lookup tables, and positions beyond the training maximum are unsupported without fine-tuning.

RoBERTa also uses learned absolute positional embeddings (same architecture as BERT). This shows that the training recipe matters more than the positional encoding scheme for many tasks.

T5 uses relative positional encoding through learned scalar biases added to attention scores. Each attention head learns a separate bias for each relative position bucket. This provides relative position awareness without changing the input embedding process.

GPT-NeoX and LLaMA use Rotary Position Embedding (RoPE), which encodes relative positions through rotation matrices applied to query and key vectors. RoPE has become the dominant choice in recent open-source language models due to its strong generalization properties.

Mistral and Mixtral also use RoPE, with additional tricks like sliding window attention to extend effective context length.

The trend in the field is clearly moving toward relative encoding schemes, particularly RoPE, for their superior generalization and extrapolation properties. However, understanding why absolute encoding was the starting point, and what its limitations are, is essential context for understanding why these newer approaches were developed.