T5 Architecture › Relative Position Biases

Standard Transformers use absolute position embeddings. Each position in the sequence gets a fixed embedding vector added to the token embedding. T5 takes a different approach with relative position biases, which encode the distance between tokens rather than their absolute locations. This design decision shows a better understanding of what position information the model needs.

Absolute positions have limitations that become apparent when you consider how humans process language. When reading the phrase "the big red ball," understanding that "big" and "red" both modify "ball" doesn't depend on whether this phrase appears at the beginning or middle of a paragraph. What matters is that these words are adjacent to each other. A model trained on sequences up to 512 tokens has never seen position 513, making generalization to longer sequences difficult with absolute positions. The embedding for position 513 simply doesn't exist, forcing awkward workarounds like position interpolation.

Relative positions encode the offset between the query and key positions in attention, capturing this linguistically real notion of proximity. Whether two words appear at positions 10 and 15 or positions 100 and 105, they're "5 positions apart" in both cases. This translation invariance helps the model generalize across different sequence positions. A relationship learned between adjacent words at the beginning of training examples automatically transfers to adjacent words anywhere in any sequence.

The key insight is that position biases should modify attention scores directly. Rather than adding position information to token embeddings (which then influences attention indirectly through the learned query/key projections), T5 adds learned biases directly to the attention logits. In standard attention, the attention score between positions $i$ and $j$ becomes:

$$\text{score}(i, j) = \frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d_k}} + b(j - i)$$

where:

• $\mathbf{q}_i$: the query vector at position $i$
• $\mathbf{k}_j$: the key vector at position $j$
• $d_k$: the dimension of the key vectors (used for scaling)
• $b(j - i)$: the learned bias for relative position $j - i$

Let's unpack what this formula tells us. The first term, $\frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d_k}}$, is the standard scaled dot-product attention score that measures content-based similarity between positions. This captures whether the semantic content at position $i$ wants to attend to the semantic content at position $j$. The second term, $b(j - i)$, adds a position-based preference that depends only on how far apart the positions are, not where they sit in the sequence. A token might learn that it generally wants to attend strongly to the immediately preceding token ($j - i = -1$), regardless of content.

The bias term $b(j - i)$ depends only on the distance between positions, not their absolute values. This allows the same position relationship to receive the same bias regardless of where it appears in the sequence. The model can learn, for example, that in English text, tokens often attend strongly to words 1-3 positions away (local syntactic patterns) while having weaker but still real attention to more distant positions (long-range dependencies).

Notice that the position bias is added directly to the pre-softmax logits, not to the post-softmax weights. This placement matters: adding to logits means the bias operates in the same space as the content-based scores, letting a large positive bias to override weak content similarity and a large negative bias to suppress attention even when content similarity is high. An absolute position embedding added to token representations, by contrast, must be routed through the learned query and key projections before influencing attention scores, giving the model less direct control over position-based attention patterns.

T5 uses a bucketed relative position scheme that balances expressiveness with parameter efficiency. Instead of learning a separate bias for every possible offset (which would require unbounded parameters as sequence length grows), T5 groups offsets into logarithmically spaced buckets. For a query at position $i$ and a key at position $j$, the relative position is computed as:

$$r = j - i$$

where:

• $r$: the relative position offset (positive when the key comes after the query, negative when before)
• $i$: the position of the query token in the sequence
• $j$: the position of the key token in the sequence

This offset $r$ is then mapped to a bucket index, and the model learns a bias value for each bucket. The bucketing strategy encodes an important insight: precise position differences matter more for nearby words than for distant ones. Whether a related word is exactly 57 or 62 positions away rarely changes its relevance, but whether it's 1 or 2 positions away often does.

The bucketing works as follows:

1. Compute the relative position: $r = j - i$ where $i$ is the query position and $j$ is the key position
2. For small offsets, use exact values (each offset gets its own bucket)
3. For larger offsets, use logarithmic bucketing (multiple offsets share a bucket)
4. Look up the learned bias for that bucket

The logarithmic spacing means nearby positions (which often carry more grammatical signal) get fine-grained distinctions, while distant positions are grouped more coarsely. This keeps the parameter count manageable while still capturing useful position information. With 32 total buckets split between forward and backward directions, the model can stand for a rich set of position relationships without requiring thousands of parameters per attention head.

For larger offsets, the bucket index is computed using logarithmic scaling:

$$b = b_{\text{exact}} + \left\lfloor \frac{\log(r / b_{\text{exact}})}{\log(d_{\text{max}} / b_{\text{exact}})} \cdot (B_{\text{dir}} - b_{\text{exact}}) \right\rfloor$$

where:

• $b$: the final bucket index for this relative position
• $b_{\text{exact}}$: the number of buckets reserved for exact (small) offsets
• $r$: the absolute value of the relative position offset
• $d_{\text{max}}$: the maximum distance considered (default 128 in T5)
• $B_{\text{dir}}$: the number of buckets per direction (half the total buckets)

This formula deserves careful examination because it reveals the design philosophy behind T5's position encoding. The numerator $\log(r / b_{\text{exact}})$ measures how far beyond the exact-bucket threshold the offset reaches, on a logarithmic scale. Dividing by $\log(d_{\text{max}} / b_{\text{exact}})$ normalizes this to a value between 0 and 1 across the range of larger offsets. Multiplying by $(B_{\text{dir}} - b_{\text{exact}})$ spreads these normalized values across the available buckets for large offsets. The floor operation ensures we get discrete bucket indices. Adding $b_{\text{exact}}$ shifts the result into the correct range, after the buckets reserved for small exact offsets.

This formula maps offsets beyond $b_{\text{exact}}$ into logarithmically-spaced buckets. This keeps the distinction between positions 1 and 2 is preserved while positions 50 and 55 share the same bucket. The logarithmic spacing means bucket boundaries grow exponentially: perhaps buckets for offsets 1, 2, 3, 4, then 5-7, 8-15, 16-31, 32-63, and so on. This mirrors human perception of distance. We notice fine distinctions between nearby objects but group distant objects more coarsely.

An important practical benefit of this design is length generalization. Because the biases are parameterized by relative distance rather than absolute position, the model can handle sequences longer than those seen during training without requiring any extrapolation of position embeddings. For offsets beyond $d_{\text{max}} = 128$, T5 simply clamps them to the last bucket, assigning the same bias to all very distant positions. This is a mild form of extrapolation, and in practice it works well: the model does not become confused by long documents, it simply treats all tokens beyond 128 positions as "distant."

Let's visualize the actual learned position biases from a trained T5 model: