Prefix Language Modeling › Key Parameters
Michael BrenndoerferUpdated July 14, 20251 min read
Language AI HandbookAll chapters Part 22 of 67Pre-training Objectives
Prefix Language Modeling › Key ParametersOne section · Sections in this chapter14 / 16 ↓
Sections in this chapter16
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
When training prefix language models, several parameters significantly affect performance:
- prefix_len: The number of tokens in the bidirectional prefix region. Determines the split point between context and generation. Longer prefixes provide richer context but leave less room for target generation.
- min_prefix_frac / max_prefix_frac: When using random prefix lengths during training (0.3 to 0.7 in our example), these control the range of possible split points. Wider ranges help the model generalize across different prefix-target ratios.
- d_model: The hidden dimension of the transformer. Larger values (64, 128, 256) increase model capacity for capturing complex patterns in both prefix and target.
- n_heads: Number of attention heads. More heads allow the model to attend to different aspects of the prefix context simultaneously.
- n_layers: Depth of the transformer stack. Deeper models can learn more complex relationships between prefix context and target generation.
- learning_rate: Typically 1e-4 to 1e-3 for prefix LM training. The bidirectional prefix may allow slightly higher learning rates than pure causal LM since gradients flow more freely.
- temperature: Controls randomness during generation (0.7 to 1.0 typical). Lower values make generation more deterministic, higher values increase diversity.