BART Architecture › Key Parameters

Understanding BART's configuration parameters is important for adapting the model to new tasks and for interpreting the behavior of different model variants. Each parameter controls a specific aspect of the architecture's capacity and behavior.

The key parameters for BART's architecture are:

• d_model: The hidden dimension size, 768 for base and 1024 for large. This is the basic width of the model: it determines the dimensionality of all token representations throughout the network. Larger values increase the model's representational capacity but proportionally increase computation and memory requirements across all layers.
• encoder_layers / decoder_layers: The number of transformer blocks in each component. For BART-base, both are 6; for BART-large, both are 12. More layers increase model capacity by letting more refinement passes over the representations, but also increase training time and inference latency proportionally to depth.
• encoder_attention_heads: The number of attention heads, 12 for base and 16 for large. Multiple heads allow the model to attend to different aspects of the input simultaneously. Each head operates on a slice of the hidden dimension ($d_k = d_{\text{model}} / \text{heads}$, so 64 dimensions per head in both base and large), letting the model to learn multiple distinct attention patterns.
• encoder_ffn_dim: The dimension of the feed-forward network's hidden layer, typically 4 times the hidden dimension (3072 for base, 4096 for large). This controls the capacity of the non-linear transformations within each block. A larger FFN dimension allows the model to represent more complex functions of the attended information.
• max_position_embeddings: The maximum sequence length the model can process, 1024 for BART. This is a hard architectural limit imposed by the learned absolute position embeddings. Attempting to process longer sequences requires extrapolation beyond the trained embedding vectors, which typically degrades performance.
• activation_function: The non-linearity used in feed-forward layers, GeLU for BART. This affects the local curvature of the loss function and the speed of training convergence.

These parameters interact in important ways. Doubling d_model roughly quadruples the computation in attention layers (due to the $O(n^2 d)$ complexity) and the computation in feed-forward layers (due to the $O(n d^2)$ complexity of the two linear projections). Doubling encoder_layers doubles the depth and thus the computation linearly. When scaling BART to larger sizes, both dimensions are typically increased together, leading to the super-linear scaling behavior that characterizes large language model training costs.