Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Learned Position Embeddings › Analyzing Real Position Embeddings

Michael BrenndoerferUpdated June 1, 20253 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 98Learned Position Embeddings › Analyzing Real Position Embeddings

Reading section 12 of 19 · Analyzing Real Position Embeddings · 3 minSection 12 of 19 · 3 min

Section 12 of 19 · 3 min read

Learned Position Embeddings › Analyzing Real Position EmbeddingsOne section · Sections in this chapter12 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When researchers analyze trained position embeddings from models like GPT-2 and BERT, several patterns emerge consistently. Understanding these empirical properties helps build intuition about what learned embeddings represent.

Low-rank structure. The 768-dimensional position embeddings can often be approximated well by a much lower-dimensional subspace. The first 50-100 principal components typically capture most of the variance. This suggests that despite living in a high-dimensional space, the effective information content of position embeddings is much lower. The extra dimensions provide capacity for task-specific adjustments during training, but the core positional structure is low-dimensional.

Smooth interpolation. Adjacent positions have similar embeddings, and this similarity decreases smoothly with distance. The embeddings form a continuous manifold in the embedding space, which is the geometric property that makes position interpolation possible.

Boundary effects. The first few positions (0, 1, 2) and positions near $L_{\max}$ sometimes show different patterns, possibly because they are encountered in distinct contexts during training. Position 0 always corresponds to special tokens like [CLS] in BERT or the beginning of a document in GPT, giving it atypical statistics. Positions near $L_{\max}$ are rare in training data if most documents are shorter than the maximum length.

Similarity to sinusoidal encodings. When you project trained position embeddings onto their principal components, the resulting patterns often look qualitatively similar to sinusoidal encodings at different frequencies. This convergence happens because sinusoidal patterns are efficient at encoding position in a way that supports the training objectives, and gradient descent discovers this even without the patterns being designed in.

Let's visualize the low-rank structure:

In[25]:

Code

from numpy.linalg import svd


def analyze_position_embedding_rank(embeddings):
    """Analyze the effective rank of position embeddings via SVD."""
    U, S, Vt = svd(embeddings, full_matrices=False)

    # Compute cumulative explained variance
    total_var = np.sum(S**2)
    cumulative_var = np.cumsum(S**2) / total_var

    return S, cumulative_var


# Analyze our simulated trained embeddings
singular_values, cumulative_var = analyze_position_embedding_rank(
    trained_embeddings
)

Out[26]:

Visualization

Line plot of singular values on log scale showing rapid exponential decay. — Singular value spectrum of position embeddings on a log scale. The rapid decay indicates that a small number of components captures most of the structural information, confirming the low effective dimensionality of learned positional representations.

Cumulative variance curve rising steeply then plateauing near 100%. — Cumulative explained variance as a function of the number of principal components. The curve reaches 90% with relatively few components, showing that the 64-dimensional position embedding space is spanned by a much smaller effective subspace.

The rapid decay of singular values confirms the low-rank structure. Most of the information in position embeddings can be captured by a handful of principal components. This suggests that positions are fundamentally simple, even though we represent them in high-dimensional space. The extra dimensions provide capacity for task-specific adjustments during training, but the core positional geometry is low-dimensional.