GPT-1 › Working with GPT-1-Era Models
Language AI HandbookAll chapters GPT-1 › Working with GPT-1-Era ModelsOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
GPT-2 Model Configuration (similar to GPT-1): Vocabulary size: 50,257 Hidden size: 768 Number of layers: 12 Number of heads: 12 Context window: 1024 Total parameters: 124,439,808
Generating Text
The pre-trained model can generate coherent text continuations:
def generate_text(prompt, max_new_tokens=20):
"""Generate text continuation using the pre-trained model (greedy decoding)."""
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
outputs = gpt2_model.generate(
inputs["input_ids"],
max_new_tokens=max_new_tokens,
do_sample=False,
pad_token_id=tokenizer.eos_token_id,
)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
return generated_textThe model generates coherent continuations because it learned to predict likely next tokens during pre-training. The quality of these completions reflects the knowledge captured from the training corpus.
Extracting Representations for Transfer
For transfer learning, we extract the hidden states at specific positions to use as input features for downstream classifiers:
def extract_representations(texts, layer=-1):
"""
Extract representations from a specific layer.
Args:
texts: List of input texts
layer: Which layer to extract from (-1 = last layer)
Returns:
Tensor of representations, shape (num_texts, hidden_size)
"""
inputs = tokenizer(
texts,
return_tensors="pt",
padding=True,
truncation=True,
max_length=512,
)
with torch.no_grad():
outputs = gpt2_model(
input_ids=inputs["input_ids"],
attention_mask=inputs["attention_mask"],
output_hidden_states=True,
)
# Get hidden states from specified layer
hidden_states = outputs.hidden_states[layer]
# Use last non-padding token's representation
seq_lengths = inputs["attention_mask"].sum(dim=1) - 1
batch_size = hidden_states.size(0)
representations = hidden_states[torch.arange(batch_size), seq_lengths]
return representationsExtracted representations shape: torch.Size([3, 768]) Representation dimension: 768 Cosine similarities between representations: Text 1 vs Text 2: 0.9777 Text 1 vs Text 3: 0.9967 Text 2 vs Text 3: 0.9669
These final-token representations contain contextual information, but their raw cosine similarities can remain uniformly high. GPT-2 was trained for next-token prediction, not to produce a calibrated sentence-embedding space. The comparison below makes that limitation visible: small relative differences exist, but a dedicated pooling strategy or sentence-level fine-tuning is needed before treating these vectors as semantic similarity scores.
Layer-wise Representations
Different layers capture different levels of abstraction. Let's visualize how representations evolve through the network:
The similarity matrix reveals the structure of information flow through the network. Early layers remain close to the embedding space, while deeper layers transform representations more dramatically. For transfer learning, intermediate layers often provide the best features because they capture generalizable linguistic patterns without becoming too specialized to the pre-training objective.