mT5 › Cross-Lingual Transfer

One of mT5's most powerful capabilities is cross-lingual transfer: the ability to fine-tune on data in one language and achieve reasonable performance in others. This property emerges from the shared multilingual representations learned during pre-training. When the model learns to predict masked spans across 101 languages simultaneously, it develops internal representations that capture language-universal patterns in how text structures information and expresses meaning.

The key insight here is that pre-training creates a shared semantic space across languages. Fine-tuning on English data teaches the model to perform a task, such as extracting answers from passages. Because the underlying representations for English and, say, Vietnamese partially overlap in this shared space, the model can apply what it learned about the task structure to Vietnamese text without ever having seen a Vietnamese training example for that task. This is not magic; it is a consequence of the model having learned, during pre-training, to encode semantically similar content in semantically similar positions in representation space, regardless of language.

Cross-lingual transfer comes in several variants depending on how much target-language data is available during fine-tuning:

• Zero-shot transfer: Fine-tune only on source language (usually English), evaluate directly on target languages
• Few-shot transfer: Fine-tune on source language, then provide a small number of target-language examples to adapt
• Translate-train: Automatically translate training data into target languages and fine-tune on the translated data
• Multilingual fine-tuning: Fine-tune on labeled data in multiple languages simultaneously when such data exists

Each strategy has different resource requirements and performance characteristics. Zero-shot transfer is the most practically useful because it requires no target-language annotations, but it also produces the largest performance gap compared to models trained on target-language data. Few-shot transfer dramatically narrows this gap with only a handful of examples.

In zero-shot transfer, a model is fine-tuned on task data in one language (typically English, where labeled data is abundant) and evaluated on the same task in other languages without seeing any target-language training examples. This capability is valuable because practitioners can use English datasets and achieve reasonable performance across dozens of languages without requiring labeled data in every language:

The visualization shows that mT5-Large achieves strong cross-lingual transfer, with Spanish and German (both related to English) reaching F1 scores above 68, while more distant languages like Hindi and Arabic show scores around 59-62. mT5 outperforms XLM-R across all languages, with improvements ranging from 5-7 F1 points. The performance gradient from English through related languages to distant languages reveals how linguistic similarity affects transfer success.

Transfer performance correlates with several factors:

• Linguistic similarity: Languages related to English (like German and Spanish) typically show better transfer than distant languages
• Script overlap: Languages sharing Latin script often transfer better due to shared subword tokens
• Pre-training data quantity: Languages with more mC4 data develop richer representations that support better transfer

Cross-lingual transfer works because mT5 learns language-agnostic representations during pre-training. The model doesn't learn 101 separate languages in isolation. Instead, it learns a unified representation space where similar concepts across languages map to similar regions, regardless of how those concepts are expressed on the surface. Several factors contribute to this alignment:

Shared vocabulary: When languages share subword tokens, especially cognates and loanwords, knowledge about these tokens transfers directly. For example, "computer" appears in similar forms in many languages. This allows the model to use what it learns about technology concepts in English text when processing Spanish or German text about the same topics. This lexical overlap creates anchor points that align representations across languages. Think of these shared tokens as bridges: when the model encounters the subword "comput" in an English text about programming, it encodes contextual meaning around that token. When it later encounters the same subword in a Spanish sentence, the representation it builds is not starting from zero. It inherits whatever the model has already learned about that token's typical contexts across all languages that share it.

Parallel structure learning: The span corruption objective forces the model to learn syntactic and semantic patterns. Many of these patterns, such as subject-verb-object ordering, generalize across languages. When the model learns that a certain span position typically contains an action word in English, this knowledge can transfer to languages with similar sentence structure. Even when word orders differ, the model learns abstract notions of "what information completes this context" that transcend specific grammatical rules. The span corruption objective is particularly well suited to encouraging this kind of structural abstraction because it requires the model to reason about what a span should contain based on surrounding context, which is at heart a semantic question rather than a lexical one.

Semantic alignment: By processing text in multiple languages about similar topics, the model learns that certain concepts are expressed similarly across languages, even when the surface forms differ. News articles about international events, Wikipedia pages about scientific concepts, and web content about popular topics appear in many languages. This provides implicit supervision for semantic alignment. This is a form of distant supervision: the model never explicitly sees pairs of translated sentences, yet the statistical regularities in multilingual web text teach it that the Spanish representation of "democratic elections" should live near the English one in representation space.

The strength of cross-lingual transfer is not uniform across all language pairs. Language pairs that are typologically similar, share script, and co-occur frequently in the same documents exhibit much stronger transfer than typologically distant pairs with unique scripts and little co-occurrence. This creates a spectrum of transfer quality across the 101 languages in mT5, from near-native performance for closely related European languages to more modest transfer for low-resource languages with distinct scripts. Understanding this spectrum helps you set realistic expectations when deploying mT5 for a given language combination.