BART Architecture › Attention Configuration

BART's attention patterns show how information flows through the model. Attention is the mechanism by which transformers route information between positions, and the pattern of allowed attention, meaning which positions can attend to which other positions, fundamentally shapes what the model can learn and compute. There are three distinct attention mechanisms in BART, each with its own configuration and purpose: bidirectional self-attention in the encoder, causal self-attention in the decoder, and cross-attention connecting the decoder to the encoder. Understanding these three patterns and why each is configured the way it is gives you a deep understanding of how information moves through the model.

Think of the attention mechanisms as a communication network within the model. Bidirectional encoder attention is like a town hall meeting where everyone can hear everyone else simultaneously. This allows rich information exchange. Causal decoder attention is like a relay race where each runner can only receive information from those who ran before. This keeps valid sequential generation. Cross-attention is like a telephone line from the decoder to the encoder. This allows the decoder to query the encoder's representations at any moment during generation. Together, these three communication patterns implement the full pipeline from input understanding to output generation.

In the encoder, every token can attend to every other token. This bidirectional attention gives the encoder the power to build contextual representations, because a word's meaning often depends on context that appears both before and after it. Consider disambiguating "The bank was eroding" versus "The bank was closing": you need both the subject and the verb to understand which sense of "bank" is intended. With bidirectional attention, the token "bank" can simultaneously look at "eroding" or "closing" to resolve this ambiguity. In a unidirectional model, "bank" would have to represent its meaning before seeing the disambiguating word that follows it.

To compute bidirectional self-attention, we construct a mask that permits all token pairs. For an input sequence of length $n$, the attention mask is an $n \times n$ matrix of ones:

$$\mathbf{M}_{\text{encoder}} = \begin{bmatrix} 1 & 1 & \cdots & 1 \\ 1 & 1 & \cdots & 1 \\ \vdots & \vdots & \ddots & \vdots \\ 1 & 1 & \cdots & 1 \end{bmatrix}$$

where:

• $\mathbf{M}_{\text{encoder}}$: the attention mask matrix of shape $n \times n$
• $n$: the length of the input sequence
• Each entry of 1 indicates that attention is permitted between that query-key pair; a 0 would block that pair

Reading this matrix, row $i$ describes which positions token $i$ can attend to: a 1 in column $j$ means token $i$ can attend to token $j$. Since every entry is 1, every token can attend to every other token, including itself. This complete connectivity allows information to flow freely through the sequence, letting the encoder to build representations that incorporate arbitrarily distant context.

Why does this formula make sense? Notice that the matrix is symmetric: if token $i$ can attend to token $j$, then token $j$ can also attend to token $i$. This symmetry reflects the bidirectional nature of the encoder: attention flows in both directions simultaneously. The uniformity of the matrix, with every entry equal to 1, reflects the fact that there are no structural constraints on encoder attention. Any token might be relevant to any other token, and the model learns through training which connections are useful.

This bidirectional attention is what gives encoder-only models like BERT their power for understanding tasks. Each token's representation is informed by the complete context, not just preceding tokens. A word at the beginning of a sentence can be influenced by words at the end, and vice versa, letting the rich contextual representations that make BERT effective for tasks like sentiment analysis and named entity recognition. BART inherits this power in its encoder, giving it the same contextual understanding capability that made BERT so successful.

The decoder uses causal (or autoregressive) masking in its self-attention layers. The word "causal" refers to the structure of language generation: each token is caused by (depends on) the tokens that came before it, not those that come after. This constraint is not artificial. During generation, future tokens do not yet exist. A model that attended to future tokens during training would be learning a task that is fundamentally different from the task it faces at inference time, where future tokens are simply unavailable.

To enforce this constraint, we use a lower-triangular mask. For a sequence of length $m$, the attention mask is:

$$\mathbf{M}_{\text{decoder}} = \begin{bmatrix} 1 & 0 & 0 & \cdots & 0 \\ 1 & 1 & 0 & \cdots & 0 \\ 1 & 1 & 1 & \cdots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & 1 & 1 & \cdots & 1 \end{bmatrix}$$

where:

• $\mathbf{M}_{\text{decoder}}$: the causal attention mask matrix of shape $m \times m$
• $m$: the length of the decoder sequence (the number of tokens generated so far)
• Entry $(i, j) = 1$ if $j \leq i$, meaning position $i$ can attend to position $j$
• Entry $(i, j) = 0$ if $j > i$, meaning position $i$ cannot attend to the future position $j$

The lower-triangular structure emerges directly from the causality constraint. Position 1 can only attend to position 1 (itself), so the first row has a single 1. Position 2 can attend to positions 1 and 2, giving two 1s. Position $i$ can attend to all positions from 1 through $i$, creating the triangular pattern. The zeros above the diagonal represent the blocked attention to future positions, which is information that the model must not use because it will not be available at generation time.

Why does this formula make sense? Consider the diagonal of the mask: every position can attend to itself, which makes sense because a token's representation should at minimum encode information about the token itself. The lower triangle means that as we move to later positions in the sequence, more context becomes available: position 1 has only itself, position 2 has itself and position 1, and so on. This is precisely the structure of autoregressive generation, where each new token is produced with access to all previously generated tokens.

This ensures that when generating token $t$, the model can only attend to tokens at positions $1, 2, \ldots, t$. This causal structure makes autoregressive generation possible. During training, we can compute attention for all positions in parallel (the mask handles the constraints), but the model learns to predict each token using only its predecessors, exactly as it will during generation. This training efficiency, combined with the discipline of causal masking, is one of the key innovations of transformer-based language models over earlier recurrent architectures.

Cross-attention connects the decoder to the encoder, bridging the understanding and generation phases. Without cross-attention, the decoder would not know what input to generate output for. It would become an unconditional language model, generating plausible text without grounding in any specific input. With cross-attention, the decoder can dynamically consult the encoder's representations, focusing on whatever parts of the input are most relevant for generating the current output token.

The mechanism of cross-attention is identical to standard scaled dot-product attention, but with a important difference: the queries come from the decoder, while the keys and values come from the encoder. This asymmetry is meaningful. The decoder is asking questions (represented as queries) about the input, and the encoder provides the answers (as values) along with ways to determine relevance (through keys). For each decoder position, queries come from the decoder hidden states while keys and values come from the encoder hidden states. There is no masking in cross-attention: every decoder position can attend to every encoder position.

We compute cross-attention as:

$$\text{CrossAttn}(\mathbf{Q}_{\text{dec}}, \mathbf{K}_{\text{enc}}, \mathbf{V}_{\text{enc}}) = \text{softmax}\left(\frac{\mathbf{Q}_{\text{dec}} \mathbf{K}_{\text{enc}}^T}{\sqrt{d_k}}\right) \mathbf{V}_{\text{enc}}$$

where:

• $\mathbf{Q}_{\text{dec}}$: query vectors derived from decoder hidden states, shape $(m, d_k)$. These represent "what the decoder is looking for" at each position.
• $\mathbf{K}_{\text{enc}}$: key vectors derived from encoder hidden states, shape $(n, d_k)$. These represent "what information each encoder position offers" and are compared against queries.
• $\mathbf{V}_{\text{enc}}$: value vectors derived from encoder hidden states, shape $(n, d_v)$. These contain the actual information that will be retrieved and combined.
• $d_k$: the dimension of query and key vectors, used for scaling
• $d_v$: the dimension of value vectors
• $\sqrt{d_k}$: a scaling factor that prevents dot products from growing too large, which would push softmax into regions with vanishing gradients

The computation proceeds in three stages. First, the dot product $\mathbf{Q}_{\text{dec}} \mathbf{K}_{\text{enc}}^T$ computes a compatibility score between each decoder position and each encoder position, resulting in an $m \times n$ matrix of raw attention scores. Second, the softmax operation converts these scaled scores into attention weights that sum to 1 across encoder positions, creating a proper probability distribution over the input that indicates how much attention each output position pays to each input position. Third, these weights are used to compute a weighted combination of encoder values, where positions with higher attention weights contribute more to the final representation.

Why does this formula make sense? The dot product $\mathbf{Q}_{\text{dec}} \mathbf{K}_{\text{enc}}^T$ measures alignment between what the decoder is looking for (queries) and what each encoder position offers (keys). High scores mean the decoder's needs align well with what that encoder position provides. After scaling and softmax, these alignment scores become a probability distribution that the decoder uses to blend together the encoder's value vectors. The result is a context vector: a weighted mixture of all encoder outputs, where the weights reflect relevance to the current decoding step.

This lets the decoder focus on different parts of the input as it generates each output token. When summarizing a document, the decoder might focus on the introduction when generating the first sentence of the summary, then shift attention to specific details when describing particular findings, and attend to the conclusion when wrapping up. This dynamic alignment is what makes encoder-decoder models so effective for conditional generation tasks, a mechanism we explored thoroughly in our discussion of Bahdanau and Luong attention in Part XII.

To understand how information flows through BART, consider processing an input sequence and generating an output. The flow follows a clear two-phase structure that separates understanding from generation while connecting them through cross-attention. The encoder and decoder do not operate completely independently: the encoder runs first, creating a complete set of hidden states, and the decoder then uses those hidden states at every layer as it generates output tokens one by one.

The encoder phase processes the entire input once. Input tokens are transformed into embeddings, positional embeddings are added, and the combined representations are passed through $L$ encoder layers. At each layer, bidirectional self-attention allows information to flow freely between all positions. By the end of the encoder, each token's representation includes context from the entire input. This phase processes the entire input once, creating a fixed set of representations that the decoder will query during generation. The computational cost of this phase is $O(n^2 d)$ where $n$ is the sequence length and $d$ is the hidden dimension, dominated by the self-attention operation.

The decoder phase generates one token at a time. For each output token, the decoder performs a sequence of operations that integrate three sources of information:

• Causal self-attention integrates information from previously generated tokens, building a representation of the output generated so far. This allows the decoder to maintain coherence across the output sequence: later tokens can attend to earlier ones. This keeps the generated text flows logically.
• Cross-attention retrieves relevant information from the encoder, grounding the generation in the input. At each step, the decoder issues queries against all encoder hidden states and receives a weighted combination of their values as a context signal.
• The feed-forward network transforms the combined representation, computing complex functions of the attended information.
• The output projection produces a probability distribution over the vocabulary, from which the next token is selected.

This two-phase structure is efficient for tasks with long inputs and shorter outputs. The expensive encoder computation happens once, and the decoder reuses those representations for every generated token. If generating a 100-word summary from a 1000-word document, the encoder runs once over 1000 tokens, and the decoder runs 100 times, each time performing cross-attention over the 1000 encoder states. The encoder's computation is amortized across all generated tokens.