Learned Position Embeddings › Key Parameters

When implementing learned position embeddings, these parameters determine the capacity and behavior of the position encoding:

• max_seq_len ($L_{\max}$): Maximum sequence length the model can handle. This determines the number of rows in the position embedding table. Common values range from 512 (BERT) to 2048 and beyond in modern GPT variants. Larger values increase memory usage linearly but allow processing longer documents without truncation. The key tradeoff is that attention cost scales quadratically with sequence length, so doubling $L_{\max}$ quadruples the attention computation.

• embed_dim ($d$): Dimension of each position embedding vector. Must match the token embedding dimension for additive combination. Typical values range from 256 to 4096 depending on model size. Higher dimensions provide more capacity but increase parameter count proportionally. In practice, the effective dimensionality of position information is much lower than $d$, so there is ample capacity even in moderate-dimension models.

• Initialization scale: Position embeddings are typically initialized with small random values, often scaled by $1/\sqrt{d}$ or using a fixed standard deviation (e.g., 0.02 in GPT-2). Smaller initialization helps training stability by keeping initial representations in a reasonable range. Very large initializations can cause position information to overwhelm token information early in training, slowing convergence.

• dropout: Dropout rate applied after combining token and position embeddings. Values of 0.1 are common. This regularizes the model by randomly zeroing embedding dimensions during training, preventing over-reliance on specific features and improving generalization.

The ratio of position parameters to total model parameters is typically small (around 2% for GPT-2), making $L_{\max}$ relatively cheap to increase from a parameter perspective. The practical bottleneck is attention complexity, which constrains how long a context the model can attend over during training and inference.