BART Architecture › The BART Encoder-Decoder Design

BART follows the encoder-decoder framework we discussed in Part XII and Part XVI, but its design philosophy draws explicitly from BERT and GPT. To appreciate this design choice, consider the basic trade-off in language modeling: understanding requires seeing the full context (including what comes before and after a word), while generation must proceed sequentially since you cannot use words you have not yet produced. These two requirements seem contradictory, yet both are needed for tasks like summarization where you must deeply understand a document before creating a coherent condensed version. A model that processes input bidirectionally can build richer representations, but it cannot generate text one token at a time, because generation requires a strictly left-to-right discipline. A model that processes input left-to-right can generate text naturally, but it builds weaker representations because each token can only incorporate information from tokens to its left.

BART resolves this tension through architectural separation. The encoder is essentially a BERT-style transformer, using bidirectional self-attention over the input sequence and letting each token to attend to all other tokens. This bidirectional view means that when the encoder processes the word "bank" in a sentence, it can simultaneously consider both the preceding context ("walked along the") and the following context ("of the river") to determine that we are discussing a riverbank rather than a financial institution. The decoder, in contrast, is essentially a GPT-style transformer, using causal (left-to-right) self-attention to ensure the model can only use previously generated tokens when predicting the next one. This constraint is not a limitation but a necessity: during generation, future tokens simply do not exist yet, and a model trained to exploit them would fail at inference time.

Think of BART's architecture as two specialist consultants working in sequence. The encoder is a research analyst who reads the entire input document with complete access to all its parts, building a complete, context-aware understanding of the content. The decoder is a writer who can consult the analyst's notes (through cross-attention) but must produce words one at a time without knowing what will come next in the output. The writer's sequential discipline ensures that what gets produced is valid text, and the analyst's complete view ensures that what the writer produces is grounded in the input. The key insight is that separating these two roles into distinct architectural components allows each to be optimized for its particular function, avoiding the compromises that would arise from forcing a single architecture to serve both purposes simultaneously.

The architecture can be summarized as:

$$\text{BART} = \text{BERT Encoder} + \text{GPT Decoder}$$

This equation describes the architecture directly. The BART authors explicitly designed the encoder to match BERT's architecture and the decoder to match GPT's, then connected them with cross-attention. The "+" here represents architectural composition: the encoder and decoder are separate components linked through a cross-attention mechanism that allows the decoder to query the encoder's representations as it generates each token.

This design means BART inherits the bidirectional contextual understanding that made BERT successful for classification and extraction tasks, while also gaining the autoregressive generation capabilities that made GPT successful for text generation. The result is a model that can first build a rich, context-aware representation of the input, then use that representation to generate fluent, coherent output. This combination is precisely what tasks such as summarization and translation require. It also supports question-answering.

The BART encoder processes the input sequence using standard transformer encoder blocks, transforming raw token embeddings into contextualized representations. Think of each encoder layer as a refinement pass: the first layer turns bare token embeddings into slightly contextualized representations, the second layer enriches those with broader context, and by the final layer each token's representation encodes a sophisticated understanding of its role within the full input sequence. The encoder performs this transformation with complete access to all input positions at every layer, a capability that distinguishes it from the decoder and that makes it capable of resolving complex ambiguities that require long-range context.

These representations capture the meaning of each token within its full surrounding context. Each block contains:

1. Multi-head self-attention with bidirectional (non-causal) masking. This is the mechanism that allows each token to "see" every other token in the input, gathering information from both directions to build context-aware representations. With multiple attention heads, the model can simultaneously attend to different aspects of the context: one head might focus on syntactic dependencies, another on semantic similarity, and another on positional proximity.

2. Feed-forward network with GeLU activation. After attention aggregates information across positions, this two-layer neural network transforms each position's representation independently, letting the model to compute complex non-linear functions of the attended information. The feed-forward network is where much of the model's factual knowledge is believed to be stored, based on analysis of what information can be retrieved from its weights.

3. Residual connections around both sublayers. These skip connections add the input of each sublayer to its output, creating direct gradient pathways that facilitate training deep networks and letting the model to learn incremental refinements rather than complete transformations. Without residual connections, gradients in deep networks tend to vanish or explode, making training unstable.

4. Layer normalization applied after each sublayer (post-norm). This normalizes the activations to have zero mean and unit variance, stabilizing training by preventing the hidden representations from growing too large or too small as they pass through many layers.

The encoder produces a sequence of hidden states, one for each input token. These states capture rich bidirectional context, as each encoder representation can incorporate information from the entire input sequence. When you pass a sentence through BART's encoder, the representation for each word includes its meaning within the sentence context. This is the foundation upon which the decoder builds its generation: rather than generating from a blank slate, the decoder starts with a complete, contextually-rich picture of the input.

The decoder generates output tokens autoregressively, one at a time, using the encoder's representations to understand the input. This process mirrors how a human might write a summary: first, you read and understand the source document (the encoder), then you compose the summary word by word while referring back to the original (the decoder with cross-attention). Each decoder block contains:

1. Causal self-attention over previously generated tokens. This allows each position to attend only to earlier positions in the output sequence, building up a representation of what has been generated so far. The causal constraint ensures the model cannot "cheat" by looking at tokens it has not yet produced. During training, the model processes the entire target sequence at once, but the mask enforces the discipline of sequential generation: each position can only see positions to its left.

2. Cross-attention over encoder hidden states. This is the bridge between understanding and generation. It allows the decoder to query the encoder's representations, focusing on different parts of the input as needed for generating each output token. Without cross-attention, the decoder would be a purely unconditional language model with no grounding in the input. With cross-attention, the decoder can dynamically focus on whatever parts of the source are most relevant to the word it is currently generating.

3. Feed-forward network with GeLU activation. Just as in the encoder, this transforms the combined self-attention and cross-attention information through a non-linear function, computing complex features of the attended representations.

4. Residual connections around all three sublayers. With three sublayers instead of two, the decoder has even more opportunity to benefit from these direct pathways that preserve information and facilitate gradient flow.

5. Layer normalization after each sublayer (post-norm). This maintains training stability across the decoder's deeper structure.

The causal masking in self-attention prevents the decoder from "cheating" by looking at future tokens during training. When training on a target sequence like "The study found significant results," the decoder predicting "found" can see "The" and "study" but not "significant" or "results." This ensures that the learned conditional distribution $P(\text{token}_t \mid \text{token}_1, \ldots, \text{token}_{t-1})$ is trained with exactly the same information available during inference. Cross-attention allows each decoder position to attend to all encoder positions, letting the decoder to ground its generation in the input context. When generating "found," the decoder can look back at the relevant parts of the source document to decide what verb is appropriate.

BART makes several design decisions that distinguish it from other encoder-decoder models. These choices reflect its heritage from BERT and GPT and affect training dynamics and model behavior in ways that practitioners should understand.

GeLU activation is the first key choice. Unlike T5, which uses ReLU in its original form, BART follows BERT and GPT-2 in using the GeLU (Gaussian Error Linear Unit) activation function in feed-forward layers. The GeLU function is defined as $\text{GeLU}(x) = x \cdot \Phi(x)$, where $\Phi(x)$ is the standard Gaussian cumulative distribution function. In practice, a common approximation is used:

$$\text{GeLU}(x) \approx 0.5 \cdot x \cdot \left(1 + \tanh\left(\sqrt{\frac{2}{\pi}} \cdot (x + 0.044715 x^3)\right)\right)$$

where:

• $x$: the pre-activation value at a given neuron
• $\Phi(x)$: the Gaussian CDF, which determines what fraction of a Gaussian distribution lies below $x$
• The product $x \cdot \Phi(x)$: scales the input by its probability under a standard normal, smoothly transitioning between near-zero for negative inputs and near-linear for positive inputs

GeLU provides smoother gradients than ReLU because it does not have a sharp transition at zero. Instead, it smoothly interpolates between passing and blocking signals. This smoothness can lead to more stable optimization and has become the preferred activation for many modern language models. Why does this formula make sense? Notice that for very negative $x$, $\Phi(x) \approx 0$, so GeLU outputs near zero, similar to ReLU. For very positive $x$, $\Phi(x) \approx 1$, so GeLU is approximately linear, also like ReLU. The difference is in the transition region near $x = 0$, where GeLU changes smoothly rather than abruptly.

Post-layer normalization is the second key choice. BART applies layer normalization after residual connections (known as post-norm), following the original transformer design. T5 uses pre-norm, applying layer normalization before each sublayer. The placement of normalization affects gradient flow: pre-norm tends to produce more stable gradients at initialization, which makes it easier to train very deep models. Post-norm can achieve slightly better final performance when training succeeds, but requires more careful tuning of learning rate schedules. This trade-off explains why both conventions persist in practice.

Learned positional embeddings are the third key choice. BART uses learned absolute position embeddings, similar to BERT and GPT-2, rather than the relative position encodings used in T5. With learned embeddings, the model maintains a separate embedding vector for each position (1, 2, 3, and so on up to some maximum), and these vectors are learned during training just like token embeddings. This approach is simple and effective but creates a hard limit on sequence length: the model has no embedding for position 1025 if it was trained with a maximum of 1024.

No parameter sharing is the fourth key choice. Unlike T5, BART does not tie encoder and decoder embeddings by default. The encoder and decoder maintain separate embedding matrices. This increases parameter count but allows the encoder and decoder to learn specialized representations suited to their different roles. The encoder's embeddings are tuned for building contextual representations of input text, while the decoder's embeddings are tuned for the language modeling objective of predicting the next token. These roles are related but distinct, and separate embedding matrices give the model the freedom to specialize each.