BART Architecture › Worked Example: Tracing a Forward Pass

To make the architecture concrete, let's trace a short input through BART's encoder and decoder step by step. This numerical walkthrough illustrates exactly how each component turns representations and where the three attention mechanisms come into play.

Suppose we want to summarize the sentence "Scientists discovered a new planet." We want BART to generate the summary "New planet found." Let's trace the processing at a conceptual level, using simplified numbers to illustrate the key transformations.

Step 1: Tokenization. The tokenizer first splits the input into BPE subword tokens. For our sentence, the result might be: <s>, Scientists, discovered, a, new, planet, ., </s>. Call this sequence of 8 tokens $\mathbf{x} = [x_1, x_2, x_3, x_4, x_5, x_6, x_7, x_8]$. Similarly, the target output tokenizes as: <s>, New, planet, found, ., </s>, giving 6 tokens $\mathbf{y} = [y_1, y_2, y_3, y_4, y_5, y_6]$.

Step 2: Input embeddings. Each input token $x_i$ is looked up in BART's embedding table to produce an embedding vector $\mathbf{e}_i \in \mathbb{R}^{768}$ (for BART-base). Then the corresponding positional embedding $\mathbf{p}_i \in \mathbb{R}^{768}$ for position $i$ is added, giving the initial encoder input:

$$\mathbf{h}_i^{(0)} = \mathbf{e}_i + \mathbf{p}_i$$

where:

• $\mathbf{h}_i^{(0)}$: the initial representation for token $i$, combining token identity and position
• $\mathbf{e}_i$: the token embedding for $x_i$, encoding the token's meaning
• $\mathbf{p}_i$: the positional embedding for position $i$, encoding the token's location in the sequence

After this step, we have 8 vectors, each of dimension 768, representing the combined token-and-position information for each input token. These are the inputs to the first encoder layer.

Step 3: Encoder processing. The 8 vectors pass through 6 encoder layers. In each layer, multi-head self-attention allows all 8 tokens to interact with each other simultaneously. Consider what happens to the token "planet" at position 6. In the first encoder layer, "planet" can already attend to "Scientists" and "discovered," picking up contextual signals that this is a scientific discovery context. In later layers, its representation becomes increasingly contextualized, incorporating information about the full sentence structure. After 6 layers, the encoder produces 8 hidden state vectors $\mathbf{h}_i^{(L)} \in \mathbb{R}^{768}$, one for each input token, where each vector encodes that token's meaning in the full sentence context.

Step 4: Decoder initialization. The decoder starts with the beginning-of-sequence token <s> as its first input. This is mapped to an embedding and positional embedding just as in the encoder, giving an initial decoder hidden state $\mathbf{d}_1^{(0)} \in \mathbb{R}^{768}$.

Step 5: Generating the first output token. The decoder processes $\mathbf{d}_1^{(0)}$ through 6 decoder layers. In each layer, three operations occur. First, causal self-attention: since we only have one decoder token so far, this simply returns the token's own representation (trivially causal with one token). Second, cross-attention: the decoder issues a query against all 8 encoder hidden states, computing compatibility scores for each. The decoder's query asks, in effect, "given that I'm about to generate the first word of the summary, which encoder states are most relevant?" The cross-attention mechanism computes a weighted combination of encoder values, likely focusing heavily on the "new planet" part of the input since that is the most content-bearing phrase. Third, the feed-forward network processes the combined representation. After all 6 layers, the decoder projects the final hidden state onto the vocabulary dimension (50,265 dimensions for BART's BPE vocabulary) and applies softmax to get a probability distribution. The highest-probability token is selected (assuming greedy decoding): "New."

Step 6: Generating subsequent tokens. The decoder now has two tokens: <s> and "New." It processes both through the decoder layers, with causal self-attention letting "New" to attend to <s> but not vice versa. Cross-attention again queries the encoder, but now from the perspective of generating the second word given that "New" was already generated. The model selects "planet" as the next token. This process continues, generating "found," ".", and finally </s>, at which point generation stops.

The key insight from this trace is that the encoder runs once and the decoder runs once per output token. For our 6-token output, the cross-attention mechanism queries the same 8 encoder hidden states 6 times, each time from a different perspective as the output sequence grows. The encoder's computation is reused across all generation steps, which is the basic efficiency advantage of the encoder-decoder architecture over pure decoder architectures for conditional generation.