Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Learned Position Embeddings › The Maximum Sequence Length Constraint

Michael BrenndoerferUpdated June 1, 20253 min read

Language AI HandbookAll chapters

Part 14 of 67Positional Encoding

Chapter 98Learned Position Embeddings › The Maximum Sequence Length Constraint

Reading section 7 of 19 · The Maximum Sequence Length Constraint · 3 minSection 7 of 19 · 3 min

Section 7 of 19 · 3 min read

Learned Position Embeddings › The Maximum Sequence Length ConstraintOne section · Sections in this chapter7 / 19 ↓

Sections in this chapter19

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Unlike sinusoidal encodings, learned position embeddings have a hard constraint: the model can only handle sequences up to length $L_{\max}$ . If you train with $L_{\max} = 512$ , you have exactly 512 position embeddings. Position 513 simply does not exist in the table.

This constraint is fundamental to the design, not a bug or an oversight. The position embedding table has exactly $L_{\max}$ rows, and there is no formula to generate additional rows. A sinusoidal encoding can compute the representation for position 10,000 even if the model was trained on sequences of length 512, because the values come from a mathematical formula that can be evaluated at any integer. Learned embeddings cannot do this: position 10,000 is simply absent from the table.

In practice, this creates several design challenges.

During training, you choose $L_{\max}$ based on your computational budget and data characteristics. Larger $L_{\max}$ means more parameters (the position table has $L_{\max} \times d$ values) and higher memory usage during training. But the more significant constraint is attention complexity: self-attention scales quadratically with sequence length, so doubling $L_{\max}$ quadruples the attention cost. For this reason, models typically choose $L_{\max}$ as the longest sequence they expect to encounter in deployment, plus a small buffer, rather than setting it to the maximum they could technically accommodate.

During inference, sequences longer than $L_{\max}$ cannot be processed directly. You must either truncate the sequence to the first $L_{\max}$ tokens (discarding the rest), use a sliding window approach that processes overlapping chunks, or extend the model through fine-tuning on longer sequences.

In[15]:

Code

# Demonstrate the sequence length constraint
max_len = 128
pos_embed_small = LearnedPositionEmbedding(max_len, 64, seed=42)


def check_position_access(pos_embed, position):
    """Check if a position can be accessed."""
    if position < pos_embed.max_seq_len:
        return True, pos_embed(np.array([position]))
    else:
        return False, None


# Test various positions
test_positions = [0, 50, 127, 128, 200]

Out[16]:

Console

Position embedding table size: 128

Position access test:
  Position   0: Valid
  Position  50: Valid
  Position 127: Valid
  Position 128: Out of range
  Position 200: Out of range

The constraint is fundamental to the approach. Sinusoidal encodings can generate a representation for any position using the formula, but learned embeddings must be stored explicitly. This is a key trade-off between the two approaches, and it motivates the extrapolation techniques discussed in the next section.

In practice, most language modeling applications work within the context window. BERT was trained with $L_{\max} = 512$ because most documents and question-answer pairs fit comfortably within 512 tokens. GPT-2 used $L_{\max} = 1024$ to accommodate longer generations. Modern models have pushed these limits to 2048, 4096, 8192, and beyond, accepting higher training costs in exchange for the ability to process longer contexts without truncation.