Transfer Learning › Types of Knowledge That Transfer

Transfer learning succeeds because pre-trained models acquire multiple types of knowledge, each useful for different downstream applications. Understanding these different knowledge types helps explain why transfer learning is so broadly effective and guides decisions about which pre-trained model to select for different tasks. The diversity of knowledge encoded in pre-trained models shows the diversity of information required to predict words accurately in natural text.

It is worth pausing to appreciate how much a language model must know to predict words accurately. Correctly predicting that the next word in "She opened the refrigerator and took out a cold" is likely "beer," "drink," "bottle," or "can" requires knowing about refrigerators, beverages, human behavior, and the compositional semantics of the surrounding phrase. The breadth of knowledge required to be a good next-word predictor is in effect the breadth of human knowledge encoded in text, and pre-trained models absorb a surprising amount of it.

The most obvious type of transfer involves core linguistic competencies. These competencies form the foundation upon which all language understanding is built, and they transfer because every language task, regardless of its specific objective, requires parsing and interpreting natural language:

• Syntax: Understanding grammatical structure, agreement patterns, and phrase boundaries. This includes knowing which words can modify which other words, how clauses nest within sentences, and how word order conveys meaning.
• Morphology: Recognizing word forms and inflections, including derivational patterns. This encompasses understanding that "running," "runs," and "ran" are forms of the same verb, and that "unhappiness" is derived from "happy" through regular morphological processes.
• Semantics: Encoding word meanings, compositional semantics, and lexical relationships. This involves knowing that "dog" and "canine" are related, that "buy" and "sell" describe the same transaction from different perspectives, and that "not unhappy" has a different meaning than "happy."
• Pragmatics: Capturing discourse structure and coherence, including communicative intent. This includes understanding that questions expect answers, that pronouns refer to previously mentioned entities, and that certain phrases signal speaker attitude or certainty.

This linguistic knowledge lets models to parse novel sentences, understand complex constructions, and handle the infinite variety of natural language. Every new sentence a model encounters differs from every sentence it saw during training, yet the model can process it because it has learned the underlying rules and patterns of the language. The compositional nature of language, which lets us understand sentences we have never seen before by combining known words according to known rules, is precisely what pre-training captures.

Pre-trained models also acquire factual knowledge about the world. Training on internet text exposes models to encyclopedic information: that Paris is the capital of France, that water freezes at $0^\circ\text{C}$, and that Einstein developed the theory of relativity. This knowledge transfers to tasks requiring factual understanding, such as question answering or fact verification.

The acquisition of world knowledge through language modeling is remarkable because the model is never explicitly told these facts. Instead, it learns them by observing patterns in how concepts co-occur. A model that sees thousands of sentences mentioning Paris in contexts involving France and its government or capital learns to associate these concepts. This implicit knowledge acquisition means that pre-trained models function as compressed databases of the information present in their training corpora.

Research has shown that larger models store more factual knowledge, explaining part of why scale improves downstream task performance. However, this knowledge can become outdated, as the model's knowledge shows its training data cutoff date. A model trained on text from 2022 will not know about events that occurred in 2023, regardless of its size. This temporal limitation is one of the inherent constraints of the pre-training approach, and it motivates techniques like retrieval-augmented generation that supplement static model knowledge with dynamic external information.

The density and reliability of world knowledge also varies across domains. Models trained on general web text will know a great deal about popular culture and history, along with widely discussed scientific topics, but may have limited and sometimes inaccurate knowledge about niche domains. For applications requiring high factual precision in specialized domains, domain-specific pre-training or explicit knowledge retrieval systems become important supplements to the base pre-trained model.

Pre-trained models also appear to learn reasoning patterns that transfer across tasks. These patterns emerge from the regularities in how humans express logical relationships in text:

• Analogical reasoning: Understanding relationships between concepts, such as knowing that Paris is to France as Berlin is to Germany
• Causal reasoning: Recognizing cause-effect relationships in text, such as understanding that "because the bridge collapsed, traffic was rerouted" indicates the collapse caused the rerouting
• Commonsense inference: Drawing everyday conclusions from context, such as inferring that someone who "grabbed an umbrella before leaving" expects rain
• Numerical reasoning: Basic arithmetic and quantitative comparisons, such as understanding that "more than half" means a majority

These abilities emerge from patterns in training text that implicitly show reasoning. A model that has seen thousands of examples explaining that "because $X$ happened, $Y$ resulted" learns to recognize causal structure even in novel contexts. This learned reasoning transfers to downstream tasks that require similar inferences, even when those tasks involve different domains or surface forms.

These reasoning patterns are statistical associations rather than true logical inference. A pre-trained model may reliably complete "if A then B; A is true, therefore..." with "B is true" not because it implements propositional logic, but because it has seen this pattern thousands of times. This distinction matters because statistical reasoning can fail on unusual or adversarial examples where the expected pattern is violated. Understanding this limitation helps calibrate expectations about when pre-trained reasoning abilities will transfer reliably.

When pre-training data includes domain-specific text, models acquire specialized knowledge. This is why domain-adapted models like BioBERT (trained on biomedical literature) or FinBERT (trained on financial text) often outperform general-purpose models on domain-specific tasks. The pre-training stage can be thought of as installing a prior over useful representations, and domain-specific pre-training installs a better prior for domain-specific tasks.

The effectiveness of domain-specific pre-training shows the fact that different domains have different vocabularies, different patterns of expression, and different background knowledge requirements. Medical text uses technical terminology and abbreviations, with writing conventions that differ from everyday language. A model pre-trained on medical literature has already learned these domain-specific patterns, which makes it better positioned to understand new medical texts than a model trained only on general web text.

Think of domain-specific pre-training as the linguistic equivalent of professional specialization. A general-purpose language model is like a highly educated generalist: knowledgeable about many subjects but not deeply expert in any specific field. A domain-specific pre-trained model is like a specialist who has read extensively in their field: they understand the vocabulary and conventions, including assumptions and reasoning patterns specific to their domain. For tasks in specialized domains, this specialist knowledge gives a significant advantage that general pre-training cannot fully replicate.