Language Identification › Language Identification

Before a language model can learn from a document, it needs to know what language that document is in. This may seem like a trivial prerequisite, but language identification sits at the foundation of nearly every modern NLP pipeline. When you are assembling a training corpus from hundreds of billions of web pages, the question of which documents belong to which language is not obvious: pages mix languages, metadata lies, encodings break, and domain-specific jargon obscures natural language boundaries.

Language identification (LangID) is the task of automatically determining the language of a piece of text. The output is typically an ISO 639 language code such as en for English, de for German, or zh for Chinese. At first glance this looks straightforward: English text looks different from Arabic text, which looks different from Japanese text. But in practice LangID must handle several challenges: multilingual documents, code-switching, very short inputs, rare languages, and adversarial noise from web scraping.

The data curation pipeline for a large language model processes enormous volumes of raw text. Filtering by language is one of the first and most impactful decisions a practitioner makes. Keep the wrong languages and you dilute signal for the languages you care about. Discard too aggressively and you starve low-resource languages of their representation. LangID is the gatekeeper, and its errors compound at scale. Misclassifying one percent of two trillion tokens means twenty billion tokens in the wrong bucket, with measurable downstream consequences for model quality.

It is worth grounding this task in history. Early computational linguists recognized that languages leave statistical traces at the character level as far back as the 1980s, and the first principled LangID systems appeared in the 1990s. The fundamental insight has not changed: every language has a characteristic distribution of character sequences. What has changed is the scale at which LangID is applied, the number of languages covered, and the robustness required for noisy web text.