Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Sinusoidal Position Encoding › Visualizing Position Encodings

Michael BrenndoerferUpdated June 6, 20257 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 97Sinusoidal Position Encoding › Visualizing Position Encodings

Reading section 4 of 15 · Visualizing Position Encodings · 7 minSection 4 of 15 · 7 min

Section 4 of 15 · 7 min read

Sinusoidal Position Encoding › Visualizing Position EncodingsOne section · Sections in this chapter4 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

With the formula and wavelength intuition in place, let's see what sinusoidal position encodings look like. We'll implement the encoding from scratch and visualize the resulting patterns to check our intuition.

In[5]:

Code

def sinusoidal_position_encoding(max_len, d_model):
    """
    Generate sinusoidal position encodings.

    Args:
        max_len: Maximum sequence length to encode
        d_model: Dimension of the encoding vectors

    Returns:
        PE: Position encoding matrix of shape (max_len, d_model)
    """
    # Create position indices: (max_len, 1)
    positions = np.arange(max_len)[:, np.newaxis]

    # Create dimension indices for pairs: (d_model/2,)
    dim_pairs = np.arange(0, d_model, 2)

    # Compute the frequency denominator: 10000^(2i/d)
    div_term = 10000 ** (dim_pairs / d_model)

    # Initialize encoding matrix
    PE = np.zeros((max_len, d_model))

    # Even dimensions: sine
    PE[:, 0::2] = np.sin(positions / div_term)

    # Odd dimensions: cosine
    PE[:, 1::2] = np.cos(positions / div_term)

    return PE


# Generate encodings for 100 positions with 64 dimensions
max_len = 100
d_model = 64
PE = sinusoidal_position_encoding(max_len, d_model)

Out[6]:

Console

Position encoding matrix shape: (100, 64)

Encoding for position 0 (first 8 dims):
  [0. 1. 0. 1. 0. 1. 0. 1.]

Encoding for position 1 (first 8 dims):
  [0.841 0.54  0.682 0.732 0.533 0.846 0.409 0.912]

Encoding for position 50 (first 8 dims):
  [-0.262  0.965 -0.203  0.979  0.157 -0.988  0.787 -0.617]

Position 0 always has sine values of 0 and cosine values of 1 in the first few dimensions, which is simply $\sin(0) = 0$ and $\cos(0) = 1$ . Notice that positions 0 and 1 look different in the high-frequency (leftmost) dimensions but will look nearly identical in the lowest-frequency dimensions, since those change so slowly. As position increases, the high-frequency dimensions (small indices) change rapidly while low-frequency dimensions (large indices) change slowly, exactly as we expect from the wavelength analysis.

Let's visualize the encoding as a heatmap to see the wave patterns:

Out[7]:

Visualization

Heatmap showing alternating light and dark bands that oscillate rapidly on the left side and slowly on the right side. — Heatmap of sinusoidal position encodings for 100 positions and 64 dimensions. Each row is a position (0-99) and each column is a dimension (0-63). The rapidly alternating bands on the left correspond to high-frequency dimension pairs that change quickly with position, while the slowly varying gradients on the right correspond to low-frequency pairs. Every row (position) is distinct. This shows that the encoding uniquely identifies each position.

The heatmap reveals the core structure. On the left side (low dimension indices), we see rapid oscillations: positions 0 and 3 might look similar here, but positions 0 and 1 are clearly different. On the right side (high dimension indices), the oscillations are so slow that the entire 100-position range barely covers a fraction of one cycle. The combination ensures every position has a unique encoding.

One detail to observe: the heatmap has a banded, almost textile-like structure. This is not noise or an artifact; it is exactly the mathematical structure we designed. The dense, narrow vertical stripes on the left represent high-frequency oscillations, and they blend into wide, gradient-like regions on the right as the wavelengths grow. If you saw a similar pattern in a completely different context, you would recognize it as a multi-frequency signal: it has the same structure as a spectrogram of a sound that contains many harmonics.

Let's examine specific dimension pairs to see the sine/cosine relationship:

Out[8]:

Visualization

Line plot showing sine and cosine waves completing about 15 cycles over 100 positions. — High-frequency encoding: dimension pair i=0 completes many cycles within 100 positions. The sine (blue) and cosine (red) waves are identical in shape but shifted by a quarter cycle. This keeps each position maps to a unique point on the unit circle.

Line plot showing sine and cosine waves completing about 2 cycles over 100 positions. — Lower-frequency encoding: dimension pair i=16 oscillates much more slowly. The same 100-position window covers far fewer cycles. This gives coarser positional discrimination that complements the high-frequency pairs.

Each dimension pair contributes a (sine, cosine) tuple that traces a circle in 2D space as position increases. The sine and cosine are 90 degrees out of phase. This ensures that every position has a unique combination even within a single dimension pair.

To make this geometric interpretation concrete, let's plot the trajectory of (sin, cos) pairs as position increases:

Out[9]:

Visualization

Scatter plot showing points tracing multiple circular loops, with color gradient indicating position. — Unit circle trajectory for the first dimension pair (i=0) across the first 50 positions. The high-frequency oscillation drives the encoding rapidly around the circle, completing multiple loops. Even though different positions may occupy similar angles, the other dimension pairs resolve any ambiguity.

Scatter plot showing points tracing a partial arc of a circle, with color gradient indicating position. — Unit circle trajectory for dimension pair i=16 across the same 50 positions. The much slower frequency means the encoding only traces a small arc. This gives coarse positional information that distinguishes positions separated by large gaps rather than adjacent steps.

The circular trajectories reveal why sine/cosine pairing works so well. In the first dimension pair (left), the fast oscillation means positions loop around the circle multiple times. Even if two positions land at similar angles on the circle, they'll be distinguished by other dimension pairs with different frequencies. In the middle dimension pair (right), positions are spread across a smaller arc. This provides coarse-grained discrimination.

Notice that in both cases, the trajectory lies exactly on the unit circle. This is a direct consequence of the identity $\sin^2(\theta) + \cos^2(\theta) = 1$ . Every position encoding, in every dimension pair, has the property that its sine and cosine values are exactly one unit apart when viewed as a 2D point. This geometric regularity is part of what makes the relative position property, which we will derive in a moment, work out so cleanly.