Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Learned Position Embeddings › Visualizing Position Embedding Geometry

Michael BrenndoerferUpdated June 1, 20252 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 98Learned Position Embeddings › Visualizing Position Embedding Geometry

Reading section 13 of 19 · Visualizing Position Embedding Geometry · 2 minSection 13 of 19 · 2 min

Section 13 of 19 · 2 min read

Learned Position Embeddings › Visualizing Position Embedding GeometryOne section · Sections in this chapter13 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The similarity analysis and singular value decomposition give us statistical summaries of position embedding structure. But we can also visualize the geometry directly by projecting position embeddings into two dimensions using principal component analysis.

In[27]:

Code

from numpy.linalg import svd as np_svd


def pca_projection(embeddings, n_components=2):
    """Project embeddings to 2D using PCA."""
    # Center the data
    centered = embeddings - embeddings.mean(axis=0)

    # SVD for PCA
    U, S, Vt = np_svd(centered, full_matrices=False)

    # Project onto top 2 components
    projected = centered @ Vt[:n_components].T
    return projected, S[:n_components] / S.sum() * 100


# Project trained embeddings to 2D
proj_2d, explained = pca_projection(trained_embeddings)

Out[28]:

Visualization

2D scatter plot of position embeddings colored by position index showing a smooth ordered curve. — PCA projection of trained position embeddings into 2D. Positions are colored from early (dark blue) to late (yellow). The smooth color gradient reveals that the embeddings form an ordered curve through the embedding space, where adjacent positions lie close together and the overall trajectory traces from beginning to end of the sequence.

The PCA visualization reveals the geometric structure of position embeddings in a way that statistics cannot. The embeddings trace a smooth curve through the two-dimensional subspace, with early positions at one end and late positions at the other. This is the geometric signature of a well-trained position embedding: positions are ordered, and the ordering is smooth. The model has learned to map the discrete sequence of positions onto a continuous manifold.

Notice that this manifold structure is exactly what makes position interpolation work. If you need to insert a new position between positions 50 and 51, you can linearly interpolate between their embeddings and land on the manifold at the right location. The interpolated point is geometrically consistent with its neighbors in the same way that any point on a smooth curve is consistent with the points around it.