Named Entity Recognition › Cross-Lingual and Multilingual NER

NER is a global problem. Text in dozens of languages needs entity extraction for practical applications, but annotated training data exists for only a small fraction of the world's languages. Cross-lingual NER addresses this gap by using high-resource language data to build models that work in low-resource languages.

The key enabler for cross-lingual NER is multilingual pre-training. Models like multilingual BERT (mBERT) and XLM-RoBERTa are pre-trained on text from 100+ languages simultaneously using a shared vocabulary and shared transformer parameters. A remarkable finding from the 2019 paper by Pires et al. is that these models develop cross-lingual representations: the representation of "Paris" in French text is close to the representation of "Paris" in English text, even though the model was never explicitly trained to align representations across languages.

This cross-lingual alignment enables zero-shot cross-lingual transfer: train a NER model on English CoNLL 2003 data, and the model generalizes to French, German, Spanish, or Dutch NER without any additional training data in those languages. In the original mBERT experiments, zero-shot transfer achieved 50-70% of fully supervised performance depending on the language, which is remarkable given that no target-language training data was used at all.

XLM-RoBERTa, trained on a larger multilingual corpus (2.5 TB of CommonCrawl data in 100 languages), substantially improves on mBERT's cross-lingual transfer performance. On the XTREME multilingual NER benchmark, XLM-RoBERTa achieves about 65 F1 in zero-shot transfer settings, compared to 58-62 F1 for mBERT.

Even with strong multilingual models, some languages pose particular challenges for NER:

Agglutinative languages like Finnish, Turkish, and Korean attach multiple morphological suffixes to a single word stem. In Finnish, "Helsingissä" (in Helsinki) and "Helsingin" (Helsinki's) are different surface forms of the same city name. Subword tokenization helps here by splitting at morpheme boundaries, but the model still needs to handle many surface variants of the same entity.

Languages without word boundaries like Chinese and Japanese require joint tokenization and NER, since there are no spaces between tokens. Chinese NER typically processes character by character, with the BIO labels assigned to characters rather than words.

Languages with grammatical case like German, Russian, and Czech change the surface form of entity mentions depending on syntactic role. "Angela Merkel" in nominative German versus "Angela Merkels" in genitive German or "Angela Merkel" in accusative German are the same entity in different grammatical forms. A model must recognize all forms as referring to the same person.

Right-to-left scripts like Arabic and Hebrew require special handling for text direction and present additional challenges because Arabic script uses connected letters with no word-internal spacing at all.

For truly low-resource languages with little or no annotated NER data, several techniques help:

Distant supervision automatically labels text by matching strings against existing knowledge bases (like Wikidata). If "Berlin" appears in Wikidata as a city, any occurrence of "Berlin" in text is automatically labeled LOC. Distant supervision is noisy (it cannot handle ambiguous names like "Springfield" or "Amazon"), but it can produce large amounts of weakly labeled training data at low cost.

Cross-lingual data augmentation translates annotated training data from a high-resource language into the target language using machine translation, then applies the original annotations to the translated text. This assumes entity mentions are preserved through translation, which is approximately true for most proper names.

Few-shot learning approaches, including prompt-based methods and retrieval-augmented methods, use small numbers of labeled examples to adapt a pre-trained model to a new domain or language. These methods are particularly relevant for NER because annotating even 50-100 sentences per entity type can significantly improve performance over zero-shot baselines.