Transfer Learning › Implications for Practice

Transfer learning has practical implications for how you approach NLP projects. The guidelines below are not rigid rules but accumulated wisdom from years of applying transfer learning across many domains and task types. They stand for starting points that you should adapt based on the specific characteristics of your problem.

Start with a pre-trained model unless a clear constraint rules it out. The burden of proof should be on training from scratch rather than using transfer learning. Even if your domain is specialized, pre-trained models provide a strong initialization that captures general linguistic competencies. The worst case for transfer learning is that the pre-trained representations are suboptimal for your domain, in which case fine-tuning will take slightly more data and epochs to converge. The worst case for training from scratch is that you never gather enough labeled data to learn the task at all.

The exceptions are narrow but real. Extremely novel domains with languages or symbol systems very different from natural text (such as protein sequence modeling or MIDI music generation) may benefit less from NLP pre-trained models. Tasks with extremely large labeled datasets (millions of examples) may not need pre-training to reach strong performance. Tasks with strict latency or memory constraints may not be compatible with large pre-trained models. Outside these cases, pre-training is usually the right starting point.

Different pre-trained models suit different tasks:

• BERT-style models: Best for classification, token labeling, and extraction tasks
• GPT-style models: Best for generation and tasks that can be framed as text completion
• T5-style models: Flexible for tasks that can be framed as text-to-text

As we will explore in upcoming chapters, the choice of base model interacts with fine-tuning strategy. Encoder-only models like BERT produce rich bidirectional representations that work well for discriminative tasks but cannot naturally generate text. Decoder-only models like GPT can generate text but see only left context, which can be limiting for tasks requiring full bidirectional understanding. Encoder-decoder models like T5 give both, at the cost of more parameters and greater complexity.

Within each architecture class, size matters. Larger models generally perform better but require more memory and compute to fine-tune. For resource-constrained applications, distilled models like DistilBERT or TinyBERT give a reasonable tradeoff: typically 40 to 60 percent smaller with only 3 to 8 percent performance degradation on most tasks. The right size depends on your computational budget and performance requirements.

If your domain differs substantially from general web text, consider domain-adaptive pre-training. Continue pre-training on domain-specific text before task-specific fine-tuning. This two-stage transfer often outperforms direct fine-tuning. The intuition is straightforward: continued pre-training on domain text helps the model learn domain-specific vocabulary and conventions, including factual associations before you ask it to learn the task-specific mapping. You are in effect giving the model additional time to build the right representational foundation before the more targeted fine-tuning step.

Domain-adaptive pre-training is particularly useful when your domain has distinctive vocabulary that may be rare in general pre-training corpora. Biomedical text is full of Latin-derived terminology and abbreviations, along with specialized notation. Legal text uses archaic constructions and highly specific terminology. Financial text has domain-specific conventions around numbers and dates, including entity names. For these domains, even a brief continued pre-training on in-domain text before fine-tuning can produce substantial improvements.

Pre-trained models reflect their training data. If your target distribution differs materially (different time period, different demographics, different register), be aware that transfer may be imperfect. Evaluate carefully and consider strategies to address distribution shift.

Distribution shift is particularly relevant when the pre-training data has known biases. Models trained on historical internet text encode the demographics and perspectives, as well as the knowledge present in that corpus. If your application serves users from underrepresented groups or handles topics underrepresented in the training data, the model's pre-trained representations may be less useful for those cases. Careful evaluation across demographic subgroups and topic domains is needed before deployment.