Transfer Learning › Summary

Transfer learning changed NLP by letting useful models trained on large unlabeled text to be adapted for specific tasks with minimal labeled data. The pre-training/fine-tuning approach separates general language understanding from task-specific adaptation. This allows the expensive work of representation learning to be amortized across many applications. By investing once in learning detailed linguistic representations, the community can reuse that investment many times across tasks and domains in many applications.

Pre-trained models acquire multiple types of knowledge that transfer: linguistic competencies encoded hierarchically across layers, world knowledge absorbed from training text, and reasoning patterns implicit in language use. This rich prior makes fine-tuning extraordinarily sample-efficient, letting strong performance from hundreds rather than hundreds of thousands of examples. The layer-by-layer analysis reveals that lower layers encode morphology and syntax while upper layers encode semantics, with each level of the hierarchy benefiting a different class of downstream tasks.

The history of transfer learning traces from static word embeddings through contextualized representations to the modern transformer-based paradigm. Each step expanded what could transfer and how effectively: from single word-level features, to contextual features for individual positions, to full end-to-end fine-tuning of deep networks that allows representations to reorganize for each task. Today, transfer learning is the default approach for nearly all NLP tasks.

However, transfer learning introduces challenges that deserve serious attention: catastrophic forgetting during fine-tuning, inherited biases from pre-training data, knowledge cutoffs that freeze models in time, and structural dependencies on a small number of model providers. In the following chapters, we will explore full fine-tuning techniques, strategies to prevent forgetting, and efficient alternatives like parameter-efficient fine-tuning that address some of these challenges while preserving the benefits that make transfer learning so powerful.