Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Learned Position Embeddings › Combining Token and Position Embeddings

Michael BrenndoerferUpdated June 1, 20253 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 98Learned Position Embeddings › Combining Token and Position Embeddings

Reading section 3 of 19 · Combining Token and Position Embeddings · 3 minSection 3 of 19 · 3 min

Section 3 of 19 · 3 min read

Learned Position Embeddings › Combining Token and Position EmbeddingsOne section · Sections in this chapter3 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Once you have position embeddings, the question is how to incorporate them into the model. The standard approach, used by virtually all transformer architectures, is simple addition: take the token embedding for each word and add the position embedding for that word's location. The result is a single vector that encodes both the identity of the token and its position in the sequence.

Just as with sinusoidal encoding, learned position embeddings are added to token embeddings:

\mathbf{h}_i = \mathbf{e}_{w_i} + \mathbf{p}_i

where:

$\mathbf{h}_i \in \mathbb{R}^d$ : the input representation for position $i$ , combining token and position information
$\mathbf{e}_{w_i} \in \mathbb{R}^d$ : the token embedding for the word at position $i$
$\mathbf{p}_i \in \mathbb{R}^d$ : the position embedding for position $i$

This additive combination means position information is blended into the token representation from the very first layer. The model sees each token as existing at a particular position, not as a position-agnostic entity. When the attention mechanism computes its QKV projections from $\mathbf{h}_i$ , those projections carry information about both what the token is and where it sits.

You might wonder whether addition is the right operation here. Why not concatenation, which would keep the two types of information separate? The answer is partly practical and partly principled. Concatenation would double the dimensionality of the representation, increasing the cost of all subsequent computations. Addition keeps the dimension fixed. More importantly, addition forces the model to learn token and position representations that work together harmoniously in the same vector space. The two sources of information must share the same dimensions, which creates pressure for them to be complementary rather than redundant.

In[7]:

Code

def combine_embeddings(token_embeddings, pos_embed):
    """
    Combine token embeddings with position embeddings.

    Args:
        token_embeddings: Token embeddings of shape (seq_len, embed_dim)
        pos_embed: LearnedPositionEmbedding instance

    Returns:
        Combined embeddings of shape (seq_len, embed_dim)
    """
    seq_len = token_embeddings.shape[0]
    positions = np.arange(seq_len)
    position_embeddings = pos_embed(positions)
    return token_embeddings + position_embeddings


# Example: combine with some token embeddings
np.random.seed(123)
token_embeds = np.random.randn(10, embed_dim) * 0.5  # 10 tokens

combined = combine_embeddings(token_embeds, pos_embed)

Out[8]:

Console

Combining token and position embeddings:
  Token embeddings shape:    (10, 64)
  Position embeddings shape: (10, 64)
  Combined shape:            (10, 64)

Example vector norms (showing position contribution):
  Position 0: token=4.709, pos=0.910, combined=4.554
  Position 5: token=3.886, pos=0.881, combined=3.958
  Position 9: token=4.019, pos=1.000, combined=4.333

The combined embeddings carry information about both what token appears at each position and where that position is in the sequence. This is the representation that attention layers will operate on. The norms show that position embeddings contribute a meaningful fraction of the total vector magnitude, confirming that positional information is not drowned out by token information.

In practice, a layer normalization step often follows the embedding addition in models like BERT, which further mixes the token and position signals before passing them to the attention layers. This normalization ensures the combined representation has a consistent scale regardless of individual embedding magnitudes.