Causal Language Modeling › Key Parameters

When training causal language models, a number of parameters significantly impact performance. Some are architectural, fixed at model design time. Others are optimization hyperparameters that you tune during training. Understanding what each controls helps you reason about tradeoffs without needing to memorize specific values.

The architectural parameters control model capacity:

• d_model: The hidden dimension of the transformer. Larger values (512, 768, 1024) increase capacity but require more compute. Our tiny model used 64. The hidden dimension determines the richness of the internal representations: a larger d_model gives the model more "space" to encode different aspects of meaning.
• n_heads: Number of attention heads. Should divide d_model evenly. More heads allow the model to attend to different aspects of context simultaneously. Each head learns a different pattern: one might track syntactic structure, another co-reference, another topical continuity.
• n_layers: Depth of the transformer stack. Deeper models can learn more complex patterns but are slower to train. Production models use 12-96 layers. Early layers tend to capture local syntactic patterns, while later layers capture more abstract semantic relationships.

The training hyperparameters control optimization:

• learning_rate: Typically 1e-4 to 6e-4 for transformers. Higher rates speed training but risk instability. Warmup schedules help stabilize early training by gradually increasing the learning rate from near zero over the first few thousand steps, allowing the model to settle into a stable optimization trajectory before taking large gradient steps.
• batch_size: Larger batches provide more stable gradients but require more memory. Modern LLMs use effective batch sizes in the millions of tokens, often achieved through gradient accumulation across many smaller physical batches when GPU memory is insufficient to hold a large batch directly.
• seq_len (context length): Maximum sequence length the model can process. Longer contexts enable better understanding but increase memory quadratically with attention. For a model with context length 4,096, each forward pass processes up to 4,096 tokens, but attention requires storing a 4,096 by 4,096 matrix per layer.

The generation-time parameter controls sampling:

• temperature: Controls randomness during generation. Values near 0 produce deterministic, repetitive output. Values near 1 produce diverse, sometimes incoherent text. Typical range: 0.7-1.0. The temperature divides the logits before the softmax: lower temperature sharpens the distribution toward the top prediction, while higher temperature flattens it, giving unlikely tokens a greater chance of being sampled.