Context Length Challenges › Training Sequence Length Limits
Language AI HandbookAll chapters Context Length Challenges › Training Sequence Length LimitsOne section · Sections in this chapter2 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Every transformer is trained on sequences of a fixed maximum length. GPT-2 used 1024 tokens. BERT used 512. Early GPT-3 models trained on 2048 tokens. These aren't arbitrary choices but rather pragmatic compromises between model capability and computational cost.
The context window (or context length) is the maximum number of tokens a transformer can process in a single forward pass. It's determined by the position embeddings used during training and the memory constraints of the hardware.
The training length fundamentally shapes what the model learns. A model trained on 512-token sequences has never seen how pronouns resolve across 1000 tokens or how conclusions connect to distant premises. These long-range patterns simply aren't in the training data. The model can't learn what it has never been shown, regardless of how powerful its architecture is in principle. This is why extending context at inference time without additional training almost always produces degraded outputs: the model is being asked to operate in a regime it has no experience with.
# Common training context lengths over time
models = [
("GPT-2", 2019, 1024),
("BERT", 2018, 512),
("GPT-3", 2020, 2048),
("LLaMA", 2023, 2048),
("LLaMA 2", 2023, 4096),
("GPT-4 Turbo", 2023, 128000),
("Claude 3", 2024, 200000),
("Gemini 1.5", 2024, 1000000),
]
# Calculate growth
years = [m[1] for m in models]
lengths = [m[2] for m in models]Evolution of Training Context Lengths: Model Year Context Length ---------------------------------------- GPT-2 2019 1,024 BERT 2018 512 GPT-3 2020 2,048 LLaMA 2023 2,048 LLaMA 2 2023 4,096 GPT-4 Turbo 2023 128K Claude 3 2024 200K Gemini 1.5 2024 1000K
The jump from 2K to 200K+ tokens represents a 100x increase in just a few years. This explosion in context length didn't come for free. Each extension required novel techniques for position encoding, memory efficiency, and training methodology.
The progression reveals a pattern. Until 2023, context lengths grew modestly, doubling every few years. Then came a breakthrough. Models jumped from 4K to 100K+ tokens in a single generation. This wasn't just better hardware. It required fundamental advances in how position information is encoded and how attention is computed.
Why Not Just Train Longer?
The obvious question: why didn't earlier models simply train on longer sequences? The answer involves multiple compounding constraints.
Memory scales quadratically. We explored this in detail in the previous chapter on the quadratic attention bottleneck. For a 4K token sequence, attention matrices alone consume 16x more memory than for 1K tokens. Training on 32K tokens with naive attention would require 1024x more attention memory than training on 1K tokens. When a single training step exhausts available GPU memory, the only options are to reduce the batch size, use gradient checkpointing, or limit the sequence length. In practice, researchers almost always chose the latter.
Compute scales quadratically too. Even if memory were free, the computational cost of training explodes. A batch of 32K-token sequences requires the same attention compute as 1024 batches of 1K-token sequences, holding total tokens constant. Training large language models already takes weeks or months on thousands of GPUs. Multiplying that cost by 1000 is simply not feasible for most organizations.
Long documents are rare. Most training data consists of short documents. Web pages average a few thousand tokens. Training a model to handle 100K tokens requires curating datasets with long documents, which are harder to find and more expensive to process. Books and academic papers provide some long-form content, but they're a small fraction of the web-scale data used for pretraining. A model trained on mostly short documents learns mostly short-range patterns.
Gradients become unstable. Backpropagating through very long sequences compounds numerical errors. Without careful normalization and training tricks, models can fail to learn anything useful from long sequences. The gradient must travel through many layers and many timesteps simultaneously, and small numerical errors multiply. This is less of a blocker than memory or compute, but it means that simply scaling up training length without architectural changes often produces poor results even when resources permit it.
def training_cost_ratio(target_length, base_length=2048):
"""
Estimate relative training cost for longer sequences.
Assumes:
- Attention scales O(n^2)
- Other components scale O(n)
- Attention dominates at long sequences
"""
attention_ratio = (target_length / base_length) ** 2
linear_ratio = target_length / base_length
# At long sequences, attention dominates (~70% of compute)
attention_weight = 0.7
linear_weight = 0.3
total_ratio = (
attention_weight * attention_ratio + linear_weight * linear_ratio
)
return total_ratio
target_lengths = [4096, 8192, 16384, 32768, 65536, 131072]
cost_ratios = [(n, training_cost_ratio(n)) for n in target_lengths]Training Cost Relative to 2048-Token Baseline:
Target Length Cost Ratio Effective Batch Reduction
------------------------------------------------------------
4K 3.4x 29.4%
8K 12.4x 8.1%
16K 47.2x 2.1%
32K 184.0x 0.5%
65K 726.4x 0.1%
131K 2886.4x 0.0%The numbers are stark. Training on 32K tokens costs roughly 180x more per sequence than training on 2K tokens. To maintain the same training throughput, you'd need 180x more compute budget. Alternatively, you accept 180x smaller effective batch sizes, which can destabilize training. Neither option is appealing.
