Transfer Learning › Worked Example: Tracing Transfer Through a Classification Task

To make the mechanics of transfer concrete, trace through what happens when you fine-tune a pre-trained BERT model for a sentiment-classification task. We will use a small example that you can verify step by step.

Suppose we have the following setup:

• A pre-trained BERT-base model with 12 transformer layers and a 768-dimensional hidden state
• A training set of 100 labeled movie reviews (50 positive, 50 negative)
• A test sentence: "The film's pacing was sluggish but the performances were transcendent."

Step 1: Tokenization and encoding. The sentence is tokenized into subword tokens: ["the", "film", "'", "s", "pac", "##ing", "was", "slug", "##gish", "but", "the", "perform", "##ances", "were", "transcend", "##ent", "."]. BERT adds [CLS] at the start and [SEP] at the end, giving 19 tokens. Each token is mapped to an integer index in BERT's 30,522-word vocabulary.

Step 2: Pre-trained forward pass (before fine-tuning). Without any fine-tuning, BERT processes these tokens through its 12 layers. The [CLS] token representation from the final layer is a 768-dimensional vector that encodes a summary of the entire sentence. At this point, the model has no idea what sentiment classification is, so this 768-dimensional vector contains rich linguistic information about the sentence but no task-specific signal.

Step 3: Fine-tuning objective. We add a linear classification head: a weight matrix $W \in \mathbb{R}^{2 \times 768}$ and a bias vector $b \in \mathbb{R}^2$ that maps the 768-dimensional [CLS] representation to 2 logits (one for positive, one for negative). The softmax of these logits gives predicted probabilities. We train this system end-to-end on our 100 labeled examples, updating both $W$ and all of BERT's parameters.

Step 4: What changes during fine-tuning. With a learning rate of $2 \times 10^{-5}$ and training for 3 epochs, the parameters change as follows. The classification head $W$ changes substantially, learning to read sentiment from the [CLS] representation. The upper layers of BERT change moderately, adjusting their representations to make sentiment signal more accessible. The lower layers of BERT change very little, because their morphological and syntactic representations are already useful for this task.

Step 5: Inference on the test sentence. After fine-tuning, BERT processes "The film's pacing was sluggish but the performances were transcendent." The model's lower layers accurately identify that "sluggish" is an adjective modifying a noun and that "transcendent" is a strong positive adjective. The middle layers parse the contrast introduced by "but," recognizing that the second clause overrides the first. The upper layers integrate these signals into a representation that the classification head reads as predominantly positive. The model correctly predicts positive sentiment despite the negative word "sluggish," because it understands the contrastive structure.

This step-by-step trace illustrates something important: the model's success on this ambiguous sentence is not due to the 100 training examples teaching it what "transcendent" means or how "but" works contrastively. It learned those things during pre-training. The 100 examples teach it only the final mapping from linguistic understanding to sentiment labels. This is the essence of transfer learning efficiency.