GPT-1 › The Architecture
Language AI HandbookAll chapters GPT-1 › The ArchitectureOne section · Sections in this chapter2 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
GPT-1 uses a decoder-only transformer architecture, a design choice that distinguished it from contemporary models like ELMo (which used bidirectional LSTMs) and from the later BERT (which used bidirectional transformer encoders). The decoder-only choice wasn't arbitrary: it enabled a simple, scalable pre-training objective based on next-token prediction. With a decoder-only architecture, the model processes text from left to right, predicting each token given all preceding tokens. This autoregressive structure means training is straightforward: run the sequence through the model, compute the probability of each token given its prefix, and minimize the cross-entropy loss. No special masking schemes, no separate encoder-decoder coupling, and no architectural asymmetry between training and inference.
The choice to use a decoder-only design, rather than a full encoder-decoder architecture like the original transformer from Vaswani et al. (2017), was deliberate. The encoder-decoder architecture excels at sequence-to-sequence tasks like translation, where the input and output have different structures. But GPT-1 aimed at language understanding tasks where the input is a text sequence and the output is a label or classification decision. A single unified decoder, applied to the reformatted input sequence, is simpler, requires fewer parameters for a given depth, and removes the architectural asymmetry that makes encoder-decoder models harder to scale uniformly. The key insight is that if you can reformulate all tasks as text sequences, you don't need the full encoder-decoder complexity.
The depth of the architecture reflects GPT-1's position as one of the first large transformer language models. At 12 layers, 768 hidden dimensions, and 12 attention heads, the model follows the same configuration that would later be used for BERT-Base. This allows direct architectural comparisons between the two approaches. Each attention head operates over a 64-dimensional subspace of the hidden representation (768 / 12 = 64), which provides enough capacity for different heads to specialize in different types of relationships. The four-fold expansion in the feed-forward network (768 to 3072 hidden dimensions) follows the convention established in the original transformer paper. This provides a bottleneck structure where the model can learn complex non-linear transformations of the attention output.
Understanding why these design choices work requires thinking about what the model needs to do during pre-training. At each position, the model reads all preceding tokens and must predict the most likely next token from a vocabulary of 40,000 subwords. Getting this right requires recognizing syntactic patterns, tracking semantic entities across long spans, resolving anaphora, and modeling discourse structure. Twelve layers of self-attention provide enough depth to compose these different types of information progressively, with early layers capturing local syntactic patterns and later layers capturing longer-range semantic and discourse relationships.
GPT-1 consists of 12 transformer decoder layers with 12 attention heads each, a hidden dimension of 768, and a context window of 512 tokens. The model contains approximately 117 million parameters.
The architectural specifications are:
| Parameter | Value |
|---|---|
| Layers | 12 |
| Hidden size () | 768 |
| Attention heads | 12 |
| Head dimension | 64 |
| Feed-forward size | 3072 |
| Context window | 512 tokens |
| Vocabulary size | 40,000 (BPE) |
| Parameters | ~117M |
These numbers match BERT-Base closely (which also has 12 layers, 768 hidden dimensions, and 12 heads), enabling direct comparisons. The key architectural difference lies in the attention pattern: GPT-1 uses causal masking where each position can only attend to previous positions, while BERT uses bidirectional attention where each position attends to the full sequence.
Transformer Decoder Stack
Each layer in GPT-1 follows the standard transformer decoder block pattern. The input passes through masked multi-head self-attention, then a position-wise feed-forward network, with residual connections and layer normalization around each sublayer. The residual connections are important: they allow gradient to flow directly back through the network without passing through the attention and feed-forward operations, which prevents the vanishing gradient problem that would otherwise make deep networks difficult to train. Layer normalization stabilizes the activations at each sublayer. This keeps the signal entering each layer has consistent scale regardless of the scale of the gradients produced by training. Together, these two techniques allow GPT-1 to be trained to depths that would have been impractical with earlier architectures.
The GPT-1 paper uses post-normalization (LayerNorm applied after the residual addition), which was the convention inherited from the original transformer. Later models, including GPT-2 and GPT-3, switched to pre-normalization (LayerNorm applied to the input before attention and feed-forward), which provides more stable training at larger scales. This seemingly minor architectural detail turns out to matter significantly at scale: pre-normalization smooths optimization, allowing higher learning rates and more reliable training convergence for very deep or very wide models.
import torch.nn as nn
class GPT1Layer(nn.Module):
"""Single transformer decoder layer as used in GPT-1."""
def __init__(
self,
hidden_size: int = 768,
num_heads: int = 12,
intermediate_size: int = 3072,
dropout: float = 0.1,
):
super().__init__()
self.attention = nn.MultiheadAttention(
embed_dim=hidden_size,
num_heads=num_heads,
dropout=dropout,
batch_first=True,
)
self.feed_forward = nn.Sequential(
nn.Linear(hidden_size, intermediate_size),
nn.GELU(),
nn.Linear(intermediate_size, hidden_size),
nn.Dropout(dropout),
)
self.norm1 = nn.LayerNorm(hidden_size)
self.norm2 = nn.LayerNorm(hidden_size)
self.dropout = nn.Dropout(dropout)
def forward(
self,
x: torch.Tensor,
attn_mask: torch.Tensor | None = None,
) -> torch.Tensor:
# Self-attention with residual (post-norm)
attn_out, _ = self.attention(x, x, x, attn_mask=attn_mask)
x = self.norm1(x + self.dropout(attn_out))
# Feed-forward with residual (post-norm)
ff_out = self.feed_forward(x)
x = self.norm2(x + ff_out)
return xGPT-1 uses GELU activation in the feed-forward network rather than ReLU. GELU provides a smoother non-linearity that empirically improves training dynamics in transformer models. The activation function multiplies each input value by the probability that a standard normal random variable would be less than that value:
where:
- : the input value to the activation function
- : the cumulative distribution function (CDF) of the standard normal distribution, which gives the probability that a standard Gaussian random variable is less than
- : the output, which smoothly transitions between suppressing small/negative values and passing large positive values
This formulation creates a smooth gating effect. For large positive , , so the output is approximately . For large negative , , so the output is approximately 0. Unlike ReLU's hard cutoff at zero, GELU provides a gradual transition that can improve gradient flow during training.
Input Representation
GPT-1's input representation combines token embeddings with learned position embeddings. Unlike the sinusoidal positional encodings from the original transformer paper, GPT-1 learns position embeddings during training. This allows the model to discover position-specific patterns relevant to its training data. Sinusoidal embeddings follow a fixed mathematical formula based on sine and cosine functions of position and dimension indices, which ensures that the model can generalize to positions it hasn't seen during training. Learned embeddings, by contrast, are just parameters initialized randomly and updated via gradient descent, which means they can specialize to the position-dependent patterns in the training data but cannot necessarily generalize to positions outside the training context window.
The decision to use learned rather than sinusoidal position embeddings was motivated by empirical performance rather than theoretical considerations. In practice, for sequences within the training context length, learned embeddings typically match or slightly outperform sinusoidal embeddings. The key practical constraint is that learned embeddings are strictly limited to the training context length: GPT-1 trained on sequences up to 512 tokens cannot use position embeddings for positions beyond 512, while sinusoidal embeddings could in principle be applied to arbitrarily long sequences. This became a significant architectural constraint as the field moved toward longer-context models.
The token embeddings themselves are 768-dimensional vectors, one per entry in the 40,000-token BPE vocabulary. These embeddings are initialized randomly and trained end-to-end alongside all other model parameters. After pre-training, these embeddings encode rich semantic and syntactic information about each token's typical linguistic context, capturing relationships that go far beyond simple word similarity. The position embeddings add a second 768-dimensional vector that encodes where in the sequence a token appears, allowing the model to distinguish between "the cat chased the dog" and "the dog chased the cat" even though both contain the same tokens in a bag-of-words representation.
class GPT1Embeddings(nn.Module):
"""Token and position embeddings for GPT-1."""
def __init__(
self,
vocab_size: int = 40000,
hidden_size: int = 768,
max_positions: int = 512,
dropout: float = 0.1,
):
super().__init__()
self.token_embeddings = nn.Embedding(vocab_size, hidden_size)
self.position_embeddings = nn.Embedding(max_positions, hidden_size)
self.dropout = nn.Dropout(dropout)
# Register position indices as buffer
self.register_buffer(
"position_ids", torch.arange(max_positions).unsqueeze(0)
)
def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
seq_len = input_ids.size(1)
position_ids = self.position_ids[:, :seq_len]
token_emb = self.token_embeddings(input_ids)
position_emb = self.position_embeddings(position_ids)
embeddings = token_emb + position_emb
return self.dropout(embeddings)Input shape: torch.Size([2, 10]) Output shape: torch.Size([2, 10, 768]) Embedding dimension: 768
The embedding layer turns input token IDs into 768-dimensional vectors. Adding position embeddings directly to token embeddings (rather than concatenating) keeps the hidden dimension constant throughout the network. This matches the original transformer design.
Causal Masking
The defining characteristic of GPT-1's decoder architecture is causal masking. During the forward pass, each token can only attend to tokens at earlier positions in the sequence. This constraint ensures that the model cannot "cheat" by looking at future tokens when predicting the next token. Think of it as the difference between reading a sentence left to right, word by word, versus reading the whole sentence and then predicting what each word "should be." The causal constraint forces the model to develop predictive representations based only on information it would realistically have access to at each position.
Without causal masking, a language model could trivially achieve perfect next-token prediction training accuracy by simply looking at the token it is supposed to predict. The mask prevents this shortcut. Every position must base its prediction purely on left context, forcing the model to learn the statistical regularities of language rather than memorizing a lookup table. This is implemented as a simple upper-triangular mask applied to the attention score matrix before the softmax: positions in the upper triangle are set to negative infinity, which causes softmax to assign them exactly zero attention weight.
The causal constraint has an important consequence for how information flows through the model. At position , the attention mechanism can gather information from any subset of positions , weighting each by relevance. This means that the model can perform long-range lookups, attending back to relevant words far earlier in the sequence, without violating the causal ordering. The multi-head structure allows different heads to perform different lookup strategies simultaneously: one head might specialize in recent context, another in syntactic dependencies, and another in thematic coherence across the full available history.
def create_causal_mask(seq_len: int) -> torch.Tensor:
"""
Create a causal attention mask.
Returns a mask where True indicates positions to block.
"""
mask = torch.triu(torch.ones(seq_len, seq_len), diagonal=1)
return mask.bool()
The triangular pattern shows that position 0 can only attend to itself, position 1 can attend to positions 0 and 1, and so on. Position 7 (the last position) can attend to all positions. This asymmetry means that later positions have access to more context, which is why language models typically generate text from left to right.
Complete Model Architecture
Assembling the components, the complete GPT-1 architecture stacks 12 decoder layers between the embedding layer and a final language modeling head. The flow through the model is: input token IDs go through the combined token and position embedding layer to produce a sequence of 768-dimensional vectors; these pass through 12 transformer decoder layers, each applying causal self-attention followed by a feed-forward network with residual connections and layer normalization; the output of the final decoder layer passes through a final layer normalization; and the resulting representations are projected to vocabulary logits via the weight-tied output matrix. The architecture is entirely sequential in the depth dimension (layer 2 can only run after layer 1 finishes) but parallel within each layer (all positions in the sequence are processed simultaneously).
Weight tying is an important detail worth understanding. The output projection layer (which maps the 768-dimensional hidden state to 40,000 vocabulary logits) uses the same weight matrix as the input token embeddings, but transposed. This means the model learns a single 40,000 × 768 matrix that is used both to map token IDs to embeddings at the input and to map hidden states back to vocabulary scores at the output. The justification is that the input and output embedding spaces should be coherent: if "cat" has a certain embedding as input, then the model should also assign a high score to "cat" at the output when its hidden state is similar to the "cat" embedding. Weight tying enforces this coherence structurally and has been consistently shown to improve performance across language model architectures.
class GPT1Model(nn.Module):
"""Complete GPT-1 architecture."""
def __init__(
self,
vocab_size: int = 40000,
hidden_size: int = 768,
num_layers: int = 12,
num_heads: int = 12,
intermediate_size: int = 3072,
max_positions: int = 512,
dropout: float = 0.1,
):
super().__init__()
self.embeddings = GPT1Embeddings(
vocab_size, hidden_size, max_positions, dropout
)
self.layers = nn.ModuleList(
[
GPT1Layer(hidden_size, num_heads, intermediate_size, dropout)
for _ in range(num_layers)
]
)
self.ln_f = nn.LayerNorm(hidden_size)
self.lm_head = nn.Linear(hidden_size, vocab_size, bias=False)
# Weight tying: share embeddings with output projection
self.lm_head.weight = self.embeddings.token_embeddings.weight
def forward(
self,
input_ids: torch.Tensor,
return_hidden_states: bool = False,
) -> dict[str, torch.Tensor]:
seq_len = input_ids.size(1)
attn_mask = create_causal_mask(seq_len).to(input_ids.device)
hidden_states = self.embeddings(input_ids)
all_hidden_states = [hidden_states] if return_hidden_states else None
for layer in self.layers:
hidden_states = layer(hidden_states, attn_mask)
if return_hidden_states:
all_hidden_states.append(hidden_states)
hidden_states = self.ln_f(hidden_states)
logits = self.lm_head(hidden_states)
output = {"logits": logits, "last_hidden_state": hidden_states}
if return_hidden_states:
output["hidden_states"] = all_hidden_states
return outputTotal parameters: 116,169,216 Input shape: torch.Size([2, 32]) Logits shape: torch.Size([2, 32, 40000]) Hidden state shape: torch.Size([2, 32, 768])
The model contains approximately 117 million parameters. Note the weight tying between the input token embeddings and the output projection layer (lm_head). This technique reduces parameters by about 30 million (40,000 vocabulary × 768 dimensions) and has been shown to improve performance by enforcing consistency between how tokens are represented at input and predicted at output.
