Language Identification › Modern Approaches: FastText-Based Models

The current standard for production LangID is the fastText library developed by Meta AI Research, with the LID.176 model covering 176 languages. Alongside this, tools like langdetect, lingua, and Google's Compact Language Detector (CLD2/CLD3) are widely used. Each makes different design decisions around speed, accuracy, language coverage, and handling of short text and multilingual content.

FastText represents each document as the average of its word and character n-gram embeddings. This is different from the probabilistic n-gram language model approach: FastText is a discriminative classifier, not a generative language model. It learns to separate languages from each other rather than modeling the probability distribution of each language independently.

For LangID specifically, the character n-gram component of FastText is particularly important. It captures morphological and orthographic patterns that are language-specific, even for out-of-vocabulary words. A word like "nieprzekraczalnych" (Polish for "impassable" in genitive plural) contains n-gram patterns that are unmistakably Polish, even if that specific word never appeared in training.

The discriminative training objective sharpens language boundaries. Consider two closely related languages like Portuguese and Spanish. A generative n-gram model trained on Portuguese will assign reasonable probability to Spanish text because the two languages overlap heavily. A discriminative classifier trained on the task of distinguishing them will learn the specific features that differentiate them, giving it a sharper decision boundary.

FastText's LID.176 model is trained on text from Wikipedia, Tatoeba, and SETimes in 176 languages. The model takes as input a short text string and produces a probability distribution over all 176 languages using softmax activation. The predicted label is the language with highest probability, and the probability value itself is a confidence score. Training is fast, inference is extremely fast (millions of documents per second on a single CPU core), and the binary model file is only about 130 MB.

Google's Compact Language Detector (CLD2, later CLD3) is another widely used system, specifically designed for web content. CLD2 is a Naïve Bayes classifier over character quadgrams with additional heuristics for scripts, character encoding, and HTML structure. CLD3 replaces the Naïve Bayes classifier with a small neural network that processes character n-grams.

CLD2 and CLD3 differ from FastText in one critical respect: they explicitly handle multilingual documents. Rather than returning a single language label, they can return a ranked list of detected languages with their estimated byte fractions, which is valuable when a page contains substantial content in multiple languages. For a data curation pipeline that wants to extract only the target-language content from mixed pages, this per-language byte fraction is directly actionable.

CLD2 also has well-calibrated confidence scores, a minimum confidence feature that suppresses unreliable predictions, and an explicit fallback to "unknown" when the input is too short or ambiguous. These properties make it suitable for pipelines that must handle the full diversity of web content.

lingua is a Python library that takes a different design philosophy: precision over coverage and speed. Where FastText covers 176 languages with high throughput but moderate accuracy on very short text, lingua focuses on fewer languages (75 as of recent versions) and uses multiple statistical models, rule-based heuristics, and language-specific knowledge to achieve higher precision on short inputs.

lingua applies a cascade of detection methods. For very short text (one to three words), it uses a list of unambiguous language-specific words. For longer text, it uses trigram and quadgram statistics. It also applies a "confidence threshold" that prevents it from making a guess below a minimum confidence level, returning an "unknown" prediction instead of a low-confidence label. This is useful in pipelines where a conservative non-prediction is better than an incorrect prediction.

The tradeoff is speed and coverage. lingua is slower than FastText by a significant margin, and it does not support most of the languages FastText covers. For a pipeline focused on a small set of languages where accuracy on short text matters more than throughput, lingua is a strong choice. For large-scale web filtering across many languages, FastText is the more practical option.

A notable recent development is OpenLID (Burchell et al., 2023), a LangID system trained on data from the GlotLID project covering over 1,600 languages. As research attention has turned to low-resource and endangered languages, the community has recognized that 176-language coverage, while impressive, excludes the vast majority of the world's languages. OpenLID addresses this gap, though its accuracy on very low-resource languages is necessarily limited by the small amount of available training data.