Encoder Architecture › Key Parameters

When implementing or configuring encoder-only transformers, several parameters directly impact model capacity, computational cost, and downstream performance:

• d_model (hidden dimension): The dimensionality of token representations throughout the encoder. BERT-Base uses 768, BERT-Large uses 1024. Larger values increase capacity but quadratically increase attention computation.

• n_layers (depth): Number of stacked encoder layers. BERT-Base uses 12, BERT-Large uses 24. More layers enable learning more complex feature hierarchies but increase memory and compute requirements linearly.

• n_heads (attention heads): Number of parallel attention heads per layer. Typically chosen so that $d_{\text{model}} / n_{\text{heads}} = 64$. More heads allow learning diverse attention patterns, but the per-head dimension decreases.

• d_ff (feed-forward dimension): Hidden dimension of the position-wise FFN. Convention is $4 \times d_{\text{model}}$. This expansion-contraction pattern provides the majority of the model's parameters.

• max_position (sequence length): Maximum number of tokens the encoder can process. BERT uses 512. Longer sequences increase memory quadratically due to attention's $O(n^2)$ complexity.

• vocab_size: Size of the tokenizer vocabulary. BERT uses ~30K WordPiece tokens, RoBERTa uses ~50K BPE tokens. Larger vocabularies reduce unknown tokens but increase embedding table size.

For fine-tuning pre-trained encoders, the key hyperparameters are learning rate (typically 1e-5 to 5e-5), batch size (16-32 for most tasks), and number of epochs (2-4 for most classification tasks).