Language Identification › Classical Approaches: N-gram Language Models

Before neural methods, the dominant approach to LangID was character-level n-gram language models. The intuition is clean: every language has a characteristic distribution of character sequences. The trigram sch is much more common in German than in English. The sequence zione is quintessentially Italian. The bigram th is a hallmark of English. The character sequence ão signals Portuguese. These distributions are stable, compact, and highly discriminative even on modest amounts of training data.

For each candidate language $L$ in a set of languages $\mathcal{L}$, we train a character-level n-gram language model on a corpus of text in that language. Given a query document $d$, we compute the log-probability of the document under each language model and classify by maximum likelihood:

$$\hat{L} = \arg\max_{L \in \mathcal{L}} \log P_L(d)$$

where:

• $\hat{L}$: the predicted language label
• $\mathcal{L}$: the set of all candidate languages
• $P_L(d)$: the probability assigned to document $d$ by the language model for language $L$
• $\log P_L(d)$: the log-probability, computed as the sum of log-probabilities of each n-gram in $d$

The log-probability of a document decomposes over its character n-grams using the chain rule:

$$\log P_L(d) = \sum_{i=1}^{|d|} \log P_L(c_i \mid c_{i-n+1}, \ldots, c_{i-1})$$

where:

• $c_i$: the $i$-th character in the document
• $P_L(c_i \mid c_{i-n+1}, \ldots, c_{i-1})$: the conditional probability of character $c_i$ given the preceding $n-1$ characters, according to language model $L$

Each language model is a lookup table of n-gram counts, normalized by the total count of all trigrams sharing the same two-character prefix. Smoothing is essential: unseen n-grams would receive zero probability under a maximum likelihood model, making the log-probability $-\infty$ for any document containing an unseen trigram. Laplace (add-one) smoothing assigns a small non-zero probability to every n-gram, even those not observed in training.

The choice of $n$ matters. Unigrams (single characters) carry almost no language-specific information beyond script identity. Bigrams start to capture language-specific patterns but are still quite general. Trigrams represent the sweet spot for most languages: they are specific enough to capture morphological signatures but common enough to appear frequently in training data. Quadgrams and longer n-grams are more discriminative but require larger training corpora to estimate reliably.

Normalization by document length is necessary when comparing documents of different lengths. A long document will accumulate a lower total log-probability simply because it is longer, not because it is less likely under any particular language model. Dividing by the number of n-grams converts absolute log-probability to average per-n-gram log-probability, making scores comparable across document lengths.

An influential alternative, introduced by Cavnar and Trenkle in 1994, avoids probability estimation entirely. Instead, it creates a ranked list of the 300 most frequent n-grams for each language. To classify a document, the algorithm computes the ranked n-gram list for the document and compares it to each language profile using a rank-distance metric called the "out-of-place" measure:

$$\Delta(d, L) = \sum_{g \in \text{top}_k(d)} \left| \text{rank}_d(g) - \text{rank}_L(g) \right|$$

where:

• $g$: a character n-gram from the document's top-$k$ most frequent n-grams
• $\text{rank}_d(g)$: the rank of n-gram $g$ in the document's frequency list
• $\text{rank}_L(g)$: the rank of n-gram $g$ in language $L$'s profile (set to a large penalty value if $g$ does not appear in the profile)

The language with the smallest out-of-place distance wins the classification. This approach is robust to short text because it depends on relative frequencies rather than absolute counts, and it does not require careful probability smoothing. The approach remains competitive for clean, well-formed text in common languages.

A word-level language model for LangID would assign probabilities based on which words appear in the document. The problem is that word-level models require a fixed vocabulary, and any word outside the vocabulary contributes nothing to classification. For a new document in a rare language, or for a document containing technical vocabulary, many words may be unknown.

Character n-grams sidestep this entirely. Every character sequence of length $n$ is a valid n-gram. A medical article in German will contain medical vocabulary that does not appear in a general German training corpus, but it will still contain German morphological patterns (the suffixes -ung, -keit, -lich, characteristic prepositions), function words spelled out character by character, and orthographic conventions like the use of ß. These signals persist regardless of domain shift.

Character n-gram models also handle agglutinative languages well. Finnish, Turkish, Hungarian, and many other languages form very long words by combining morphemes, meaning that word-level models face severe vocabulary explosion while character n-gram models see stable distributions.