Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Position Problem › Position Encoding vs. Position Embedding

Michael BrenndoerferUpdated June 5, 20256 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 96Position Problem › Position Encoding vs. Position Embedding

Reading section 5 of 11 · Position Encoding vs. Position Embedding · 6 minSection 5 of 11 · 6 min

Section 5 of 11 · 6 min read

Position Problem › Position Encoding vs. Position EmbeddingOne section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Before diving into specific solutions, let us clarify an important terminological distinction that causes persistent confusion in the literature.

Encoding vs. Embedding

Positional encoding refers to a fixed, deterministic function that maps position indices to vectors. The function is defined by a mathematical formula, not by training. Positional embedding refers to learned vectors stored in a lookup table, one per position. Both inject positional information into the model, but through fundamentally different mechanisms with different trade-offs.

Positional encoding uses a predetermined formula to compute a position vector. Given position $p$ , the encoding function $f(p)$ returns a fixed vector that never changes during training. No gradient flows through the positional encoding itself; it is a fixed input to the model. The classic example is the sinusoidal encoding from the original Transformer paper, which uses sine and cosine functions at different frequencies to construct a unique vector for each position. The primary advantage is generalization: because the formula is defined for all integers, it applies to any position, including those longer than anything seen during training. The model can, in principle, handle sequences of arbitrary length. The drawback is that the encoding is fixed and may not capture the positional patterns most useful for a specific task.

Positional embedding treats positions like vocabulary items. We create an embedding matrix $\mathbf{P} \in \mathbb{R}^{L \times d}$ , where $L$ is the maximum sequence length and $d$ is the embedding dimension. Position $p$ looks up row $p$ of this matrix, retrieving a $d$ -dimensional learned vector $\mathbf{p}_p$ . These embeddings are initialized randomly and then updated through training by gradient descent, just like word embeddings. The advantage is flexibility: the model can discover whatever positional representations are useful given the training data and task. The disadvantages are that positions beyond $L$ have no representation, and the embeddings must be trained from scratch, requiring sufficient examples of each position to learn useful representations.

Most modern language models use positional embeddings (learned lookup tables) for short-to-medium contexts. Models designed for longer contexts or better generalization tend to use more sophisticated positional schemes like RoPE or ALiBi. The terminology gets mixed in the literature, with both approaches sometimes loosely called "positional encoding," so it is worth being precise about the distinction.

How Position Information Enters the ModelLink Copied

Now that we understand the two mechanisms for generating positional vectors, the remaining question is: how do we inject this information into the attention computation?

Recall from our earlier analysis that attention weights depend only on query and key vectors, which are projections of the input embeddings. To make attention position-aware, we need to modify what goes into those projections. There are several conceptual approaches:

Input modification: Change the token embeddings before they enter the model, so that the Q/K/V projections see both content and position
Attention score modification: Directly add a positional bias to the attention scores after the Q/K dot product but before softmax
Relative encoding in projections: Modify the key vectors to include relative position terms

The standard approach in the original Transformer (and in most BERT-style models) is option 1: add the positional vector directly to the token embedding, creating a single combined representation that carries both semantic meaning and positional information:

\mathbf{x}'_i = \mathbf{x}_i + \mathbf{p}_i

where:

$\mathbf{x}_i \in \mathbb{R}^d$ : the token embedding at position $i$ (carries semantic content)
$\mathbf{p}_i \in \mathbb{R}^d$ : the positional vector for position $i$ (carries positional information)
$\mathbf{x}'_i \in \mathbb{R}^d$ : the combined representation fed to the self-attention layers
$d$ : the embedding dimension (must match for both token and position vectors, enabling element-wise addition)

Why addition rather than concatenation or multiplication? Addition is the simplest operation that preserves dimensionality. If we concatenated positional vectors, the combined representation would have dimension $2d$ , requiring all downstream weight matrices to be twice as wide. Addition keeps everything at dimension $d$ . More subtly, addition allows the model to learn how much weight to give positional versus semantic information through the QKV projection matrices. If the model finds certain dimensions of the positional encoding uninformative for a task, the learned projection matrices can simply down-weight those dimensions. The addition operation is also differentiable, so gradients flow through it cleanly during training.

In practice, the positional vectors are often scaled to have a similar magnitude to the token embeddings, to prevent either one from dominating the combined representation. This is particularly important with sinusoidal encodings, which we will examine in the next chapter.

In[11]:

Code

def add_positional_info(token_embeddings, position_vectors):
    """
    Combine token embeddings with positional information.

    Args:
        token_embeddings: Shape (seq_len, embed_dim)
        position_vectors: Shape (seq_len, embed_dim)

    Returns:
        Combined representations of shape (seq_len, embed_dim)
    """
    return token_embeddings + position_vectors


# Example: simple positional vectors (not a real encoding scheme)
seq_len_ex = 4
embed_dim_ex = 8

# Token embeddings
tokens_ex = np.random.randn(seq_len_ex, embed_dim_ex)

# Placeholder positional vectors (small scale to illustrate the combination)
positions_ex = np.random.randn(seq_len_ex, embed_dim_ex) * 0.1

# Combined representation
combined_ex = add_positional_info(tokens_ex, positions_ex)

Out[12]:

Console

Shapes:
  Token embeddings:     (4, 8)
  Position vectors:     (4, 8)
  Combined:             (4, 8)

Example: Position 0
  Token:    [[-1.254 -0.638  0.907 -1.429]...]
  Position: [[0.089 0.175 0.15  0.107]...]
  Combined: [[-1.165 -0.462  1.057 -1.322]...]

The shapes confirm that adding positional vectors preserves the embedding dimension. Looking at position 0, you can see how each dimension of the combined vector is simply the sum of the corresponding token and position dimensions. The positional vectors are scaled by 0.1 here to keep them relatively small compared to the token embeddings. This is a common practice that prevents positional information from dominating semantic content at the start of training. Over time, the model can adjust the influence of positional information through its learned projections.

The combined representation now carries both semantic content (from the token embedding) and positional context (from the positional vector). When this combined representation is projected into QKV representations, both types of information influence the attention pattern. A token's query vector asks "what am I looking for, given my content and position?" A token's key vector offers "what do I provide, given my content and position?" The dot product between these position-aware queries and keys produces attention weights that reflect both semantic relevance and positional relationships.

The Alternative: Modifying Attention Scores DirectlyLink Copied

Some approaches, including ALiBi (which we will cover in a later chapter), bypass input modification entirely and instead add position-dependent biases directly to the attention scores:

\alpha_{ij} = \text{softmax}\!\left(\frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d_k}} + b(i, j)\right)

where $b(i, j)$ is a bias term that depends on the positions $i$ and $j$ (typically just their distance $|i - j|$ ). This approach has the advantage that the position signal never gets mixed with the semantic content in the embeddings. Semantic relationships are computed purely from content, and then positional biases are applied afterward to adjust attention patterns. This cleaner separation can improve extrapolation: the semantic attention mechanism is unchanged, and positional biases are applied on top.

Understanding that there are multiple points of entry for positional information will be important as we survey positional encoding methods in subsequent chapters. Different methods inject position at different stages of the computation, leading to different trade-offs.