Transformer Architecture (2017) › The Solution

The Transformer architecture solved these problems by eliminating recurrence entirely and relying solely on attention mechanisms. The key insight was that attention could connect encoder and decoder states and enable all positions in a sequence to attend to all other positions simultaneously. This self-attention mechanism would allow parallel processing during training while providing direct connections between any two positions in the sequence, regardless of distance.

The core innovation of the Transformer was self-attention, a mechanism that allows each position in a sequence to attend to all positions in the same sequence, including itself. Unlike the attention mechanisms in encoder-decoder models that connected different sequences, self-attention operates within a single sequence, enabling the model to capture relationships between any pair of positions directly.

Self-attention computes a representation for each position by taking a weighted combination of all positions in the sequence, where the weights are determined by how relevant each position is to the current position. The mechanism uses three learned linear transformations to create query, key, and value vectors from the input representations. For each position, the query vector represents what information is being sought, the key vectors from all positions represent what information is available, and the value vectors contain the actual information to be aggregated.

The attention scores are computed by taking the dot product between the query vector at one position and the key vectors at all positions, including itself. These scores are then scaled by the square root of the dimension to prevent the dot products from growing too large, and a softmax function normalizes the scores into attention weights that sum to one. The final representation for each position is computed by taking a weighted sum of the value vectors, where the weights are the attention weights. This process allows each position to selectively focus on the most relevant information from all other positions in the sequence.

The mathematical formulation of self-attention begins with input representations $X$ where each row corresponds to a position in the sequence. The model learns three weight matrices $W_Q$, $W_K$, and $W_V$ that transform the input into query, key, and value matrices:

$$Q = X W_Q, \quad K = X W_K, \quad V = X W_V$$

The attention scores are computed as scaled dot-products between queries and keys:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

where $d_k$ is the dimension of the key vectors. The scaling factor $\sqrt{d_k}$ prevents the dot products from becoming too large when $d_k$ is large, which would push the softmax into regions where it has extremely small gradients. The softmax ensures that attention weights sum to one, creating a probability distribution over positions. The final output combines value vectors weighted by these attention weights, allowing each position to incorporate information from all other positions in proportion to their relevance.

The Transformer uses multi-head attention, which applies the self-attention mechanism multiple times in parallel with different learned transformations. Instead of performing a single attention computation, the model learns multiple sets of query, key, and value transformations, allowing it to attend to information from different representation subspaces simultaneously. Each attention computation is called a "head," and the outputs from all heads are concatenated and then linearly transformed to produce the final output.

Multi-head attention enables the model to capture different types of relationships simultaneously. One head might learn to attend to syntactic relationships, focusing on grammatical connections between words. Another head might attend to semantic relationships, identifying words that are conceptually related. Yet another head might attend to long-range dependencies, connecting information from distant parts of the sequence. Combining these attention patterns produces richer representations than a single attention mechanism can provide.

The multi-head attention mechanism first applies multiple sets of learned linear transformations to create multiple query, key, and value matrices. Each head computes attention independently using its own set of transformations. The outputs from all heads are concatenated and then projected through a final linear transformation. This design allows the model to specialize different heads for different types of relationships while maintaining computational efficiency through parallel computation of all heads.

Since the Transformer processes all positions in parallel and doesn't use recurrence, it lacks any inherent sense of sequence order. To address this, the architecture adds positional encodings to the input embeddings. This provided information about the position of each element in the sequence. These positional encodings are added element-wise to the input embeddings before they enter the first transformer layer, allowing the model to incorporate sequence order information.

The original Transformer used fixed sinusoidal positional encodings that varied according to a mathematical pattern. For each position $pos$ and dimension $i$, the encoding was computed using sine and cosine functions with different frequencies. For even dimensions, the encoding used sine functions, and for odd dimensions, it used cosine functions. This pattern creates unique encodings for each position that the model can learn to interpret, while the mathematical structure allows the model to generalize to sequence lengths beyond those seen during training.

The sinusoidal approach was chosen because it enables the model to learn relative positions, as the encoding for position $pos + k$ can be represented as a linear function of the encoding for position $pos$. This property allows the model to understand that the relationship between positions is consistent, even when processing sequences of different lengths. While many modern transformer implementations use learned positional embeddings instead of fixed sinusoidal encodings, the core idea of explicitly encoding positional information remains essential to the architecture.

The Transformer uses an encoder-decoder architecture, where both the encoder and decoder are stacks of identical layers. The encoder consists of a stack of encoder layers, each containing a multi-head self-attention mechanism followed by a position-wise feed-forward network. Residual connections around each sub-layer and layer normalization ensure stable training and enable the model to learn effectively despite the deep architecture.

The encoder processes the input sequence, creating representations that capture relationships within the input. Each encoder layer refines these representations, allowing the model to build increasingly abstract and contextualized understanding of the input. The encoder's output is a rich representation of the input sequence that the decoder can use during generation.

The decoder also consists of a stack of decoder layers, but with a more complex structure. Each decoder layer contains three sub-layers: masked multi-head self-attention, multi-head attention over the encoder output, and a position-wise feed-forward network. The masked self-attention prevents the decoder from attending to future positions during training. This ensured that generation proceeds left-to-right without leaking information about future tokens. The encoder-decoder attention connects the decoder to the encoder output, allowing the decoder to focus on relevant parts of the input when generating each output token.

All sub-layers use residual connections and layer normalization. Residual connections enable gradients to flow directly through the network, addressing the vanishing gradient problem that can occur in deep networks. Layer normalization stabilizes training by normalizing activations within each layer, reducing internal covariate shift and allowing for larger learning rates and more stable optimization.

The feed-forward networks in each layer are simple two-layer neural networks that transform each position independently. These networks typically expand the representation to a higher dimension, apply a non-linear activation function, and then project back to the original dimension. This design allows the model to perform complex transformations on each position's representation after it has been updated by attention mechanisms.