Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Sinusoidal Position Encoding › Worked Example: Encoding a Short Sentence

Michael BrenndoerferUpdated June 6, 20254 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 97Sinusoidal Position Encoding › Worked Example: Encoding a Short Sentence

Reading section 6 of 15 · Worked Example: Encoding a Short Sentence · 4 minSection 6 of 15 · 4 min

Section 6 of 15 · 4 min read

Sinusoidal Position Encoding › Worked Example: Encoding a Short SentenceOne section · Sections in this chapter6 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's make the abstract concrete with a small, worked example. Consider a sentence with five tokens: "The cat sat on mat." We'll encode the positions 0 through 4 using a tiny encoding with $d = 8$ dimensions, small enough to trace through by hand for the first few values.

With $d = 8$ , we have $d/2 = 4$ dimension pairs, numbered $i = 0, 1, 2, 3$ . The frequency denominators for each pair are:

\begin{aligned} i = 0: \quad 10000^{0/8} &= 10000^{0} = 1 \\ i = 1: \quad 10000^{2/8} &= 10000^{0.25} \approx 17.78 \\ i = 2: \quad 10000^{4/8} &= 10000^{0.5} = 100 \\ i = 3: \quad 10000^{6/8} &= 10000^{0.75} \approx 562.3 \end{aligned}

The wavelengths for each pair are:

Pair 0: $\lambda = 2\pi \cdot 1 \approx 6.28$ positions
Pair 1: $\lambda = 2\pi \cdot 17.78 \approx 111.7$ positions
Pair 2: $\lambda = 2\pi \cdot 100 \approx 628$ positions
Pair 3: $\lambda = 2\pi \cdot 562.3 \approx 3532$ positions

For position 0 (the token "The"), every sine term evaluates to $\sin(0) = 0$ and every cosine term evaluates to $\cos(0) = 1$ , giving:

PE_0 = [0, 1, 0, 1, 0, 1, 0, 1]

This is a special property of position 0 that makes it immediately recognizable: it is the only position where the odd dimensions (cosines) are all 1.

For position 1 (the token "cat"), we compute:

\begin{aligned} PE_{1,0} &= \sin(1/1) = \sin(1) \approx 0.841 \\ PE_{1,1} &= \cos(1/1) = \cos(1) \approx 0.540 \\ PE_{1,2} &= \sin(1/17.78) \approx \sin(0.0562) \approx 0.056 \\ PE_{1,3} &= \cos(1/17.78) \approx \cos(0.0562) \approx 0.998 \\ PE_{1,4} &= \sin(1/100) = \sin(0.01) \approx 0.010 \\ PE_{1,5} &= \cos(1/100) = \cos(0.01) \approx 1.000 \\ PE_{1,6} &= \sin(1/562.3) \approx \sin(0.00178) \approx 0.002 \\ PE_{1,7} &= \cos(1/562.3) \approx \cos(0.00178) \approx 1.000 \end{aligned}

So $PE_1 \approx [0.841, 0.540, 0.056, 0.998, 0.010, 1.000, 0.002, 1.000]$ .

The pattern is clear: in the first dimension pair ( $i = 0$ ), there is already a large change from position 0 to position 1 (from $[0, 1]$ to $[0.841, 0.540]$ ), because the wavelength is only about 6. In the later dimension pairs ( $i = 2$ and $i = 3$ ), the change is tiny, because a single step is a tiny fraction of their 628- or 3532-position wavelengths.

Now consider position 4 (the token "mat") in the first dimension pair: $\sin(4) \approx -0.757$ and $\cos(4) \approx -0.654$ . This is very different from both position 0 and position 1, confirming that the high-frequency pair rapidly distinguishes even nearby positions. Meanwhile, the slow-frequency pairs at $i = 2$ and $i = 3$ have barely moved from their position-0 values, since 4 is much less than the wavelengths of 628 and 3532.

In[13]:

Code

# Worked example: encode a 5-token sequence with d=8
d_example = 8
tokens = ["The", "cat", "sat", "on", "mat"]
PE_example = sinusoidal_position_encoding(len(tokens), d_example)

Out[14]:

Console

Position encodings for 'The cat sat on mat' (d=8)
============================================================
Token    Pos   dim0  dim1  dim2  dim3  dim4  dim5  dim6  dim7
------------------------------------------------------------
The      0     +0.000  +1.000  +0.000  +1.000  +0.000  +1.000  +0.000  +1.000
cat      1     +0.841  +0.540  +0.100  +0.995  +0.010  +1.000  +0.001  +1.000
sat      2     +0.909  -0.416  +0.199  +0.980  +0.020  +1.000  +0.002  +1.000
on       3     +0.141  -0.990  +0.296  +0.955  +0.030  +1.000  +0.003  +1.000
mat      4     -0.757  -0.654  +0.389  +0.921  +0.040  +0.999  +0.004  +1.000

Out[15]:

Visualization

Small heatmap showing 5 rows and 8 columns of position encoding values for a short example sequence. — Position encodings for a 5-token sequence with d=8 dimensions. Each row is one token position and each column is one dimension. Notice that position 0 ('The') has all-zero sine values and all-one cosine values. High-frequency dimensions (left) change substantially between adjacent positions, while low-frequency dimensions (right) barely change across the entire 5-token sequence.

This worked example makes the abstract formula tangible. You can trace exactly how each token's position influences its encoding, and you can see that the information is spread across all dimensions simultaneously rather than concentrated in any single one. In practice, these position encodings are added element-wise to the token embeddings, so the final input to the transformer is the sum of what the token means (its semantic embedding) and where it sits in the sequence (its positional encoding).