Relative Position Encoding › In Practice: Adopting Relative Position Encoding

Shaw et al.'s formulation has been adopted in several widely-used models, and understanding how those adoptions differ from the original paper helps calibrate when and how to use relative position encoding in your own work.

BERT and GPT variants originally used absolute position embeddings (either sinusoidal or learned), but researchers have since explored retrofitting relative position attention to these architectures. The main finding is that relative encoding consistently outperforms absolute encoding on tasks requiring cross-sentence reasoning or document-level understanding, where relevant information can appear at variable distances.

T5 uses a simplified form of relative position encoding that omits the value-side embeddings entirely and uses a smaller number of learned biases shared across all heads. This shared-bias design reduces the parameter count and works well for text-to-text generation tasks. T5's position biases are learned per-layer but shared across heads, a further simplification that incurs little performance cost.

Transformer-XL introduced segment-level recurrence to handle very long sequences, and paired it with a relative position encoding scheme based on sinusoidal functions rather than learned embeddings. Their formulation is more complex than Shaw et al.'s but avoids the need to re-initialize position embeddings when extending to new sequence lengths.

When implementing relative position encoding in practice, several configuration decisions matter:

• Clipping distance $k$: For sentence-level tasks, $k = 16$ or $k = 32$ is usually sufficient. For document-level tasks or tasks with long-range dependencies (coreference resolution, discourse structure), larger values like $k = 64$ or $k = 128$ are more appropriate.

• Value-side embeddings: Omitting $\mathbf{a}^V$ reduces parameters by half with modest performance loss. For memory-constrained settings or when training data is limited, this is a reasonable trade-off.

• Initialization scale: Relative position embeddings should be initialized with smaller values than QKV projections. A scale of $1/\sqrt{d_k}$ or the Xavier formula used in the code above prevents position biases from dominating attention scores at the start of training.

• Multi-head sharing: Some implementations share position embeddings across all heads (T5's approach), while others use per-head embeddings (Shaw et al.'s approach). Shared embeddings reduce parameters but limit each head's ability to specialize in different distance profiles.

The most important practical consideration is batching. In standard attention, you batch multiple sequences together and process them in parallel. With relative position encoding, all sequences in a batch share the same position embedding table, but the offset matrix for each sequence needs to be computed based on that sequence's length. This is straightforward for fixed-length sequences (pad to the same length) but requires careful handling for variable-length sequences with attention masking.