Position Encoding Comparison › Key Parameters

When implementing position encoding methods, several parameters significantly influence behavior and performance. Understanding these helps you configure each method appropriately for your use case.

• max_len / max_position_embeddings: Maximum sequence length the model can handle. For learned embeddings, this is a hard limit. For formula-based methods, it determines pre-computation range but doesn't restrict extrapolation.

• embed_dim / d_model: Dimension of position vectors. Must match the token embedding dimension for additive methods (sinusoidal, learned). For RoPE, determines the number of rotation frequency pairs.

• base (default: 10000): Controls the wavelength range. Higher values create longer wavelengths, spreading position information across more positions. The original Transformer used 10000, which works well for sequences up to several thousand tokens.

• Initialization scale: Random initialization magnitude affects training dynamics. Small values (0.01-0.02) prevent position information from dominating early training.

• base (default: 10000): Same role as sinusoidal, controlling frequency decay across dimensions. Lower values create faster-varying patterns, potentially better for short-range dependencies.

• scaling_factor: For position interpolation, determines how positions are compressed. A factor of 2.0 compresses 8K positions into the 0-4K range the model saw during training.

• num_heads: Number of attention heads determines the slope distribution. More heads provide finer-grained distance sensitivity across different attention patterns.

• Slope computation: Slopes follow a geometric sequence from $2^{-8/n}$ to $2^{-8}$, where $n$ is the number of heads. Lower slopes (closer to 0) create gentler distance penalties for some heads.

• alpha (NTK interpolation): Scaling factor for the frequency base. Values around 1.5-2.0 typically work well for 2-4x context extension.

• scale (linear interpolation): Ratio of training length to target length. Computed as train_len / target_len.