Learned Position Embeddings › Initialization and Training Dynamics
Language AI HandbookAll chapters Learned Position Embeddings › Initialization and Training DynamicsOne section · Sections in this chapter9 / 19 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The initialization of position embeddings deserves careful attention because it affects how quickly they learn meaningful structure. Several initialization strategies have been used in practice.
The most common approach, used in GPT-2, initializes position embeddings from a normal distribution with standard deviation 0.02. This small value keeps the initial representations in a range where the network activations are well-behaved: not so small that position information vanishes in the noise, not so large that it dominates the token embeddings and destabilizes early training.
Another approach, sometimes used in models that inherit architecture from Word2Vec-style pretraining, initializes position embeddings with the same distribution as token embeddings. This treats position as just another feature to embed, with no special consideration for its unique properties. In practice, both approaches work similarly because training quickly overrides the initialization.
The training dynamics of position embeddings differ from other parameters in one subtle way: positions are sampled non-uniformly during training. In most text corpora, short sequences are more common than long ones. A model trained on Wikipedia articles will see many examples of positions 0 through 50 but relatively few examples of positions 400 through 512. This means later positions receive weaker gradient signals and may learn less reliable representations. This sampling bias can cause models to perform slightly worse on text that uses the full context window compared to text that fits comfortably in a shorter window.
In practice, the gradient signal is strong enough for the positions that appear most frequently in the training data, and those are typically the positions that matter most in deployment. Positions near the end of the context window are also positions where the model has the most context to work with, so their representations may need to distinguish among more possible contexts. The potential weakness in their training is partially offset by the fact that gradient signals from attention over long contexts provide rich learning signal when long sequences do occur.