Context Length Challenges › Key Parameters

When analyzing context length limitations or designing systems that work with long sequences, these parameters directly determine feasibility and performance:

• n (sequence length): The number of tokens in the input. This is the most critical parameter since both memory and compute scale with $n^2$ for attention. Typical values range from 512 (early BERT) to 1M+ (modern long-context models).

• h (number of attention heads): Each head stores an independent $n \times n$ attention matrix. Memory scales linearly with head count. Common values: 12 (GPT-2 Small) to 128 (large models).

• L (number of layers): Transformer depth. Both attention memory and KV cache scale linearly with layer count. Typical range: 12-96 layers.

• d (model dimension): The embedding size. KV cache scales as $O(L \times n \times d)$. Common values: 768 (BERT-base) to 8192 (large LLMs).

• dtype_bytes: Precision of stored values. Using fp16/bf16 (2 bytes) instead of fp32 (4 bytes) halves memory requirements. Most modern training uses bf16.

• training_length: The maximum sequence length seen during training. Determines the "safe" operating range for position encodings. Extrapolation beyond this length causes degradation unless mitigation techniques are applied.

• base (RoPE): The base constant in rotary position embeddings (default 10000). Affects the frequency spectrum of position encoding. Modifying this is one approach to extending context (NTK-aware scaling).

• window_size (sliding window attention): For models using local attention, this determines how many tokens each position can attend to. Larger windows capture more context but use more memory. Common values: 256-4096 tokens.