Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Learned Position Embeddings › The Position Embedding Table

Michael BrenndoerferUpdated June 1, 20256 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 98Learned Position Embeddings › The Position Embedding Table

Reading section 2 of 19 · The Position Embedding Table · 6 minSection 2 of 19 · 6 min

Section 2 of 19 · 6 min read

Learned Position Embeddings › The Position Embedding TableOne section · Sections in this chapter2 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The core idea is simple enough to state in one sentence: maintain a lookup table of position vectors, one for each position in the sequence, and when processing a sequence, retrieve the embedding for each position and add it to the corresponding token embedding. This sentence contains the entire mechanism. Everything else is elaboration on why it works, what the learned vectors look like, and what constraints the approach imposes.

Think of the position embedding table as an analog to the token embedding table. When a transformer processes text, it first converts each token into a dense vector by looking up that token in the vocabulary embedding matrix. This matrix has one row per token in the vocabulary, and each row is a learnable parameter vector. The position embedding table works identically: it has one row per position up to the maximum sequence length, and each row is a learnable parameter vector. The only difference is what gets indexed. Token embeddings are indexed by token identity (which word is this?), while position embeddings are indexed by position (where is this word in the sequence?).

Position Embedding Table

A position embedding table is a trainable matrix of shape $(L_{\max}, d)$ , where $L_{\max}$ is the maximum sequence length and $d$ is the embedding dimension. Position $i$ is represented by the $i$ -th row of this matrix. These vectors are learned during training, not computed from a formula.

Mathematically, if $\mathbf{P} \in \mathbb{R}^{L_{\max} \times d}$ is the position embedding table, then the position embedding for position $i$ is:

\mathbf{p}_i = \mathbf{P}[i, :]

where:

$\mathbf{p}_i \in \mathbb{R}^d$ : the position embedding vector for position $i$
$\mathbf{P} \in \mathbb{R}^{L_{\max} \times d}$ : the position embedding table (a learnable parameter matrix)
$L_{\max}$ : the maximum sequence length the model can handle
$d$ : the embedding dimension (same as token embeddings)
$i$ : the position index (0-indexed, so $i \in \{0, 1, \ldots, L_{\max} - 1\}$ )

This is identical to how word embeddings work: just as we look up a word's embedding from a vocabulary table, we look up a position's embedding from a position table. The key insight is that the lookup operation is differentiable with respect to the table entries, so gradients can flow through the lookup and update the position vectors during training.

Notice that this formula contains no mathematical structure about position. The vector $\mathbf{p}_5$ has no a priori relationship to $\mathbf{p}_6$ . The vectors start as random noise and only acquire structure through training. This is both the strength and the weakness of the approach: the model is free to discover any positional structure, but it must discover it from scratch using training data.

In[3]:

Code

import matplotlib.pyplot as plt  # noqa: F401
import numpy as np


class LearnedPositionEmbedding:
    """
    Learned position embedding table.

    This is a simplified version that stores the embedding table as a NumPy array.
    In practice, this would be a PyTorch nn.Embedding layer.
    """

    def __init__(self, max_seq_len, embed_dim, seed=None):
        """
        Initialize the position embedding table.

        Args:
            max_seq_len: Maximum sequence length (L_max)
            embed_dim: Embedding dimension (d)
            seed: Random seed for reproducibility
        """
        if seed is not None:
            np.random.seed(seed)

        # Initialize with small random values (like word embeddings)
        # Using normal initialization scaled by 1/sqrt(embed_dim)
        self.embeddings = np.random.randn(max_seq_len, embed_dim) * (
            1.0 / np.sqrt(embed_dim)
        )
        self.max_seq_len = max_seq_len
        self.embed_dim = embed_dim

    def __call__(self, positions):
        """
        Look up position embeddings.

        Args:
            positions: Array of position indices

        Returns:
            Position embeddings of shape (len(positions), embed_dim)
        """
        return self.embeddings[positions]

The implementation mirrors how word embedding layers work in deep learning frameworks. In PyTorch, you would use nn.Embedding(max_seq_len, embed_dim), which handles the lookup and gradient computation automatically.

Let's create a position embedding table and examine what the initial (random) embeddings look like:

In[4]:

Code

# Create a position embedding table
max_seq_len = 128
embed_dim = 64

pos_embed = LearnedPositionEmbedding(max_seq_len, embed_dim, seed=42)

# Look up embeddings for positions 0 through 9
positions = np.arange(10)
pos_vectors = pos_embed(positions)

Out[5]:

Console

Position embedding table shape: (128, 64)
  - 128 positions
  - 64 dimensions per position

Sample position embeddings (first 5 dimensions):
  Position 0: [ 0.062 -0.017  0.081  0.19  -0.029] ...
  Position 1: [ 0.102  0.17  -0.009  0.125  0.045] ...
  Position 2: [ 0.012 -0.063 -0.194  0.009 -0.133] ...
  Position 3: [ 0.027 -0.156  0.022  0.048 -0.11 ] ...
  Position 4: [ 0.158 -0.088  0.055  0.097 -0.116] ...

At initialization, the position embeddings are random vectors with no meaningful structure. The magic happens during training, when gradients flow back through these embeddings and reshape them to capture positional patterns useful for the task. The initialization values are small (scaled by $1/\sqrt{d}$ ) to keep the initial representations in a reasonable range that avoids saturating activations early in training.

Out[6]:

Visualization

Heatmap of random position embeddings showing no coherent structure or patterns. — Random position embeddings at initialization. Each row represents a dimension and each column a position. The lack of any vertical structure shows that adjacent positions have completely unrelated embeddings at the start of training. Gradient descent will transform this noise into smooth, meaningful positional patterns over the course of training.

The heatmap reveals the chaotic structure of random initialization. Each column (position) has values that appear unrelated to neighboring columns. There is no smooth gradient, no pattern that would help the model understand that position 5 is closer to position 6 than to position 50. This randomness is the starting point. Training will transform this noise into meaningful structure through the pressure of minimizing prediction loss.