Language Identification › Visualizations
Language AI HandbookAll chapters Language Identification › VisualizationsOne section · Sections in this chapter6 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's visualize key aspects of language identification: how character trigram distributions differ across languages, how confidence varies with document length, and the tradeoff between coverage and precision at different confidence thresholds.
Character N-gram Profiles
The trigram profiles make the LangID signal visually tangible. Each language has a different ranked pattern, but the individual entries also reflect the vocabulary and subject matter in this deliberately small training corpus. A production system therefore learns from far more varied text and compares the complete distribution across hundreds or thousands of trigrams rather than relying on a few prominent fragments.
Confidence vs. Document Length
A critical property of LangID systems is that short-text scores are often unstable. The raw score margin used by this toy model is not a calibrated probability, so plotting it by document length also reveals where a seemingly intuitive confidence proxy can mislead.
# Generate test sentences of varying lengths by taking prefixes of longer texts
full_sentences = {
"en": "the transformer model processes sequences by computing attention weights between all pairs of tokens in the input enabling parallel computation and long range dependencies",
"fr": "le modele transformateur traite les sequences en calculant des poids d attention entre toutes les paires de tokens dans l entree permettant un calcul parallele et des dependances a longue portee",
"de": "das transformer modell verarbeitet sequenzen durch berechnung von aufmerksamkeitsgewichten zwischen allen tokenpaaren in der eingabe was parallele berechnung und weitreichende abhaengigkeiten ermoeglicht",
}
# Test at different word counts
word_counts = [1, 2, 3, 5, 7, 10, 15, 20, 25, 30]
confidence_by_length = {lang: [] for lang in full_sentences}
for lang, full_text in full_sentences.items():
words = full_text.split()
for wc in word_counts:
snippet = " ".join(words[: min(wc, len(words))])
predicted, scores = predict_language(snippet, ngram_models)
sorted_scores = sorted(scores.values(), reverse=True)
if len(sorted_scores) >= 2:
# Margin between top-1 and top-2 as a proxy for confidence
margin = sorted_scores[0] - sorted_scores[1]
conf = min(
1.0, max(0.0, (margin + 2.0) / 4.0)
) # normalize to [0,1]
else:
conf = 1.0
confidence_by_length[lang].append(conf)
The plot shows why the score margin should not be interpreted directly as confidence. The model averages log probabilities by the number of trigrams, so longer prefixes do not automatically accumulate a larger margin. A short but distinctive fragment can appear spuriously decisive, while longer prefixes converge toward different language-specific levels.
A minimum document length can still be a useful operational rule, but it must be validated against a labeled set. Production pipelines typically calibrate model scores and measure accuracy by length bucket before choosing separate thresholds for short and long documents.
Confusion Matrix Across Languages
Let's evaluate the n-gram model on all five languages and visualize the confusion matrix to understand which languages are most often confused with each other.
# Build held-out test set: take one sentence from each language not in training
HELD_OUT = {
"en": [
"the model generates text by predicting the next token at each step",
"pretraining on large corpora gives models broad language understanding",
"fine tuning adapts the pretrained model to specific downstream tasks",
"the vocabulary is built using byte pair encoding to handle rare words",
],
"fr": [
"le modele genere du texte en predisant le prochain token a chaque etape",
"le preentrainement sur de grands corpus donne aux modeles une large comprehension du langage",
"l ajustement fin adapte le modele preentraine a des taches specifiques en aval",
"le vocabulaire est construit en utilisant le codage par paires d octets pour gerer les mots rares",
],
"de": [
"das modell generiert text indem es bei jedem schritt das naechste token vorhersagt",
"das vortraining auf grossen korpora gibt modellen ein breites sprachverstaendnis",
"die feinabstimmung passt das vortrainierte modell an spezifische nachgelagerte aufgaben an",
"das vokabular wird mit byte pair encoding erstellt um seltene woerter zu behandeln",
],
"es": [
"el modelo genera texto prediciendo el siguiente token en cada paso",
"el preentrenamiento en grandes corpus le da a los modelos una amplia comprension del lenguaje",
"el ajuste fino adapta el modelo preentrenado a tareas especificas posteriores",
"el vocabulario se construye utilizando codificacion de pares de bytes para manejar palabras raras",
],
"it": [
"il modello genera testo prevedendo il prossimo token ad ogni passo",
"il preaddestramento su grandi corpus fornisce ai modelli una comprensione ampia del linguaggio",
"la messa a punto adatta il modello preaddestrato a compiti specifici a valle",
"il vocabolario viene costruito utilizzando la codifica a coppie di byte per gestire le parole rare",
],
}
LANG_LIST = ["en", "fr", "de", "es", "it"]
confusion = np.zeros((5, 5), dtype=int)
for i, true_lang in enumerate(LANG_LIST):
for sentence in HELD_OUT[true_lang]:
predicted, _ = predict_language(sentence, ngram_models)
j = LANG_LIST.index(predicted)
confusion[i][j] += 1
# Accuracy per language
per_lang_acc = [
confusion[i][i] / confusion[i].sum() if confusion[i].sum() > 0 else 0.0
for i in range(5)
]
overall_acc = confusion.diagonal().sum() / confusion.sum()
Overall accuracy: 100.0% Per-language accuracy: en: 100.0% fr: 100.0% de: 100.0% es: 100.0% it: 100.0%
The matrix is perfectly diagonal because this held-out set contains only four clean, topic-matched sentences per language. That makes it useful as an execution sanity check, but far too small and homogeneous for estimating real error rates or identifying the hardest language pairs.
A meaningful evaluation would include many more documents, varied domains, short and noisy inputs, and closely related language varieties. Larger benchmarks often expose weaker separation between related languages, but this particular matrix does not show such errors and should not be used to claim that it does.
Confidence Threshold vs. Precision and Coverage
In data curation, you must choose a confidence threshold that balances two competing goals: maximizing the precision of your language-filtered corpus (keeping only correctly identified documents) while maximizing coverage (retaining as many documents as possible). A higher threshold gives you more reliable language predictions but discards more documents. Let's visualize this tradeoff directly.
# Simulate scores for a large set of documents
# We model two populations: correct predictions (high confidence) and wrong predictions (lower confidence)
np.random.seed(42)
n_docs = 2000
# True positive scores: correctly classified docs tend to have high confidence
tp_scores = np.clip(np.random.beta(8, 2, size=int(n_docs * 0.85)), 0.01, 0.999)
# False positive scores: incorrectly classified docs tend to have lower but sometimes high confidence
fp_scores = np.clip(np.random.beta(3, 5, size=int(n_docs * 0.15)), 0.01, 0.999)
all_scores = np.concatenate([tp_scores, fp_scores])
all_labels = np.array([True] * len(tp_scores) + [False] * len(fp_scores))
thresholds = np.linspace(0.3, 0.99, 50)
precision_vals = []
coverage_vals = []
for thresh in thresholds:
accepted = all_scores >= thresh
if accepted.sum() == 0:
precision_vals.append(1.0)
coverage_vals.append(0.0)
else:
precision_vals.append(all_labels[accepted].mean())
coverage_vals.append(accepted.sum() / len(all_scores))
The useful operating zone is better identified by the bend in the coverage curve than by the curves' crossing point, which occurs near the low end of this simulation. Precision is already high at moderate thresholds, while coverage begins falling much faster beyond roughly 0.7. The 0.7 to 0.9 range therefore illustrates progressively stronger corpus-purity choices with increasingly large losses in retained documents.
In practice, the optimal threshold depends on your use case. For pretraining data where you have 500 billion tokens to work with, you can afford aggressive filtering and might choose 0.9. For a low-resource language where every correct document is precious, you might choose 0.5 and accept more noise in exchange for higher recall.