Decoder Architecture › Key Parameters

When configuring a decoder-only transformer, the following parameters have the greatest impact on model capacity and performance:

• d_model (model dimension): The dimensionality of token representations throughout the network. Common values range from 256 (small models) to 12288 (GPT-3 scale). Must be divisible by n_heads. Larger values increase capacity but quadratically increase attention computation.

• n_heads (number of attention heads): How many parallel attention patterns the model can learn. Typically set so that d_model / n_heads yields head dimensions of 64-128. More heads enable diverse attention patterns but have diminishing returns beyond a point.

• n_layers (number of decoder blocks): The depth of the transformer stack. Deeper models learn more abstract representations. GPT-2 Small uses 12 layers; GPT-3 uses 96. Depth trades off against width for a given parameter budget.

• d_ff (feed-forward hidden dimension): The expansion dimension in the FFN sublayer. Traditionally 4x d_model, but modern models use 2.7x-3.5x with gated activations like SwiGLU. Larger FFN dimensions increase the network's non-linear transformation capacity.

• vocab_size: The number of tokens in the vocabulary. Larger vocabularies reduce sequence length for the same text but increase embedding and LM head parameters. Typical values range from 30K (GPT-2) to 128K (Llama 3).

• max_seq_len: Maximum sequence length the model can process. Affects position embedding size and memory requirements during training. Modern models support 2K-128K tokens through efficient attention mechanisms.

• temperature (generation): Controls randomness during sampling. Values below 1.0 sharpen the distribution (more deterministic); values above 1.0 flatten it (more random). Common range is 0.7-1.0 for generation tasks.