Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Learned Position Embeddings › Position Similarity Analysis

Michael BrenndoerferUpdated June 1, 20254 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 98Learned Position Embeddings › Position Similarity Analysis

Reading section 6 of 19 · Position Similarity Analysis · 4 minSection 6 of 19 · 4 min

Section 6 of 19 · 4 min read

Learned Position Embeddings › Position Similarity AnalysisOne section · Sections in this chapter6 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

One way to understand what position embeddings have learned is to compute similarities between positions. If the model uses position for word order and syntax, nearby positions should be more similar than distant ones.

The cosine similarity between two position embeddings $\mathbf{p}_i$ and $\mathbf{p}_j$ measures how aligned their directions are in the embedding space:

\text{sim}(i, j) = \frac{\mathbf{p}_i \cdot \mathbf{p}_j}{\|\mathbf{p}_i\| \cdot \|\mathbf{p}_j\|}

where:

$\mathbf{p}_i \cdot \mathbf{p}_j$ : the dot product of position embeddings $i$ and $j$
$\|\mathbf{p}_i\|$ : the Euclidean norm of position embedding $i$
$\text{sim}(i, j) \in [-1, 1]$ : a value of 1 means identical direction, 0 means orthogonal, -1 means opposite direction

When we compute this similarity for all pairs of positions and visualize the result as a matrix, we get a window into the geometric structure that training has imposed on the position embeddings. If positions are organized smoothly, we expect a band of high similarity along the diagonal, gradually fading to lower values as the distance between positions increases.

In[11]:

Code

def cosine_similarity_matrix(embeddings):
    """Compute pairwise cosine similarities between embeddings."""
    # Normalize each embedding to unit length
    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
    normalized = embeddings / (norms + 1e-8)

    # Cosine similarity = dot product of normalized vectors
    return normalized @ normalized.T


# Compute similarity for trained embeddings
sim_matrix = cosine_similarity_matrix(trained_embeddings)

Out[12]:

Visualization

Heatmap showing cosine similarity between positions with a bright diagonal and gradually decreasing values off-diagonal. — Cosine similarity between position embeddings. The bright diagonal confirms each position is most similar to itself. The gradual fade off the diagonal shows that nearby positions have more similar embeddings than distant ones, a property that emerges from training and enables the model to distinguish local from distant context.

The similarity matrix reveals the structure learned by position embeddings. The bright diagonal indicates that each position is most similar to itself (similarity = 1). The gradual darkening as we move away from the diagonal shows that nearby positions are more similar than distant ones. This band structure is characteristic of well-trained position embeddings. The pattern differs from sinusoidal encodings, which produce a more regular, wave-like similarity structure. Learned embeddings develop their own characteristic shape based on how positions are used in the training data.

Let's quantify how similarity decreases with distance:

In[13]:

Code

def similarity_by_distance(sim_matrix, max_distance=30):
    """Compute average similarity as a function of position distance."""
    n = sim_matrix.shape[0]
    distances = []
    avg_similarities = []

    for d in range(max_distance + 1):
        sims = []
        for i in range(n - d):
            sims.append(sim_matrix[i, i + d])
        distances.append(d)
        avg_similarities.append(np.mean(sims))

    return np.array(distances), np.array(avg_similarities)


distances, avg_sims = similarity_by_distance(sim_matrix, max_distance=40)

Out[14]:

Visualization

Line plot showing cosine similarity decreasing from 1.0 at distance 0 to near 0 at distance 40. — Average cosine similarity between position embeddings as a function of positional distance. Similarity starts at 1.0 for distance 0 (each position compared to itself) and decays smoothly toward 0 as distance increases. This decay profile is the geometric signature of a well-trained position embedding: the model has learned to map nearby positions to similar vectors and distant positions to unrelated vectors.

The decay curve shows that similarity drops off smoothly with distance. Positions 1 apart have high similarity (around 0.9), while positions 20 or more apart have near-zero similarity. This pattern allows the model to detect relative position through embedding similarity: if two position embeddings are very similar, the positions are likely close together. Notice that this relative position information emerges implicitly from absolute position embeddings. The model does not explicitly store "distance 3" as a concept; it learns absolute positions whose geometry happens to encode relative distance as a byproduct.