Relative Position Encoding › Key Parameters

When implementing relative position encoding, several parameters control the behavior and efficiency of the mechanism. Getting these right matters more for relative encoding than for absolute encoding, because the position embeddings are learned and their training dynamics depend on initialization and capacity choices.

The key parameters are:

• max_distance (k): Maximum relative position to distinguish. Positions beyond this distance share the same "far away" embedding. Typical values range from 16 to 128. Smaller values reduce memory but lose fine-grained distance information for distant positions. Larger values preserve more detail but require more parameters and training data. The right value depends on the task: sentence-level tasks rarely need $k > 32$, while document-level tasks may benefit from $k = 128$.

• d_k: Dimension of query and key vectors, and key-side relative position embeddings. Must match between queries and keys for dot product compatibility. Common values are 64 for single-head attention or embed_dim / num_heads for multi-head attention. Larger $d_k$ gives the position embeddings more capacity to encode complex distance patterns.

• d_v: Dimension of value vectors and value-side relative position embeddings. Determines the output dimension of each attention head. Often set equal to d_k for simplicity, though they can differ.

• Embedding initialization scale: Relative position embeddings are typically initialized with small random values. Xavier/Glorot initialization scales by $\sqrt{2/(n\_\text{positions} + d)}$ to maintain stable gradient magnitudes during training. If initialization is too large, position biases dominate the attention scores from the start and the model struggles to learn content-based patterns. If too small, position information takes many epochs to develop.

The total number of relative position parameters is $(2k + 1) \times (d_k + d_v)$. With $k = 64$ and $d_k = d_v = 64$, this adds 16,512 parameters per attention head, which is modest compared to the QKV projection matrices (which have $3 \times d_{\text{model}} \times d_k$ parameters). In a 12-head transformer with $d_{\text{model}} = 768$, the QKV projections total about 1.77M parameters per layer, while relative position embeddings add only about 200K. The overhead is real but small.