Rotary Position Embedding (RoPE) › Key Parameters

When implementing RoPE in your models, a small number of parameters control the behavior of the rotation. Understanding what each one does makes it easier to adapt RoPE for non-standard architectures or longer contexts.

When implementing RoPE in your models, these parameters control its behavior:

• d_model (embedding dimension): The total dimension of query and key vectors. Must be even since RoPE operates on dimension pairs. Common values range from 64 to 4096, typically matching the model's hidden dimension divided by the number of attention heads.

• base (frequency base): Controls the range of rotation frequencies. The default value of 10000 provides a good balance between local and global position sensitivity. Larger values (e.g., 100000) extend the effective context length by slowing all rotations; smaller values make the encoding more sensitive to nearby positions.

• theta_i (per-dimension frequency): Computed as $1/\text{base}^{2i/d}$ for dimension pair $i$. Not typically set directly, but understanding this helps diagnose behavior: the first pair rotates once per radian, while the last pair completes a full rotation over approximately $2\pi \times \text{base}$ positions.

• Position offset: Some implementations support a starting position offset for key-value caching during inference. This allows continuing generation from a specific position without recomputing RoPE for all previous positions.

Choosing the right base value is the most consequential decision when deploying RoPE. The original value of 10,000 works well for sequences up to about 4,000 tokens. For 32,000-token contexts, values around 100,000 to 500,000 are common. For contexts of a million tokens or more, researchers have explored dynamic base scaling, where the effective base is adjusted at inference time based on the actual input length. These extensions preserve the core rotation mechanism while stretching the frequency range to cover longer distances without aliasing.