Hallucination Causes › Training Data Issues

The simplest explanation for why models say false things is that they learned from false things. A language model is, in one sense, a compressed statistical summary of its training corpus. If that corpus contains errors, the model will reproduce them. But the problem runs deeper than occasional inaccuracies. The structure of web-scale training data introduces systematic biases that affect entire categories of knowledge, not just isolated facts.

Modern large language models are trained on enormous web crawls: CommonCrawl, C4, The Pile, and similar datasets containing hundreds of billions to trillions of tokens. Web scraping at this scale is unavoidable given the data requirements of large models, but the web is not a curated encyclopedia. It contains:

• Factual errors in forums and blogs that are stated confidently and never corrected
• Outdated information that was accurate when written but no longer reflects the world
• Duplicate and near-duplicate content that artificially overweights certain claims
• Spam, SEO content, and boilerplate with low informational density but high token volume
• Hallucinated content from prior AI systems, increasingly common as generated text propagates and gets indexed

When a model trains on this data, it learns to produce text in the statistical style of the corpus, errors included. The model has no mechanism for distinguishing "this website stated X confidently" from "X is true." Both contribute equally to the optimization objective.

The training objective is next-token prediction: for a corpus of tokens $x_1, x_2, \ldots, x_N$, the model maximizes:

$$\mathcal{L}_{\text{LM}} = \sum_{t=1}^{N} \log P(x_t \mid x_1, x_2, \ldots, x_{t-1})$$

where:

• $\mathcal{L}_{\text{LM}}$: the total language modeling loss, summed over all $N$ token positions in the corpus
• $x_t$: the token at position $t$ (the target token to predict)
• $x_1, x_2, \ldots, x_{t-1}$: all preceding tokens, forming the context the model conditions on
• $P(x_t \mid x_1, \ldots, x_{t-1})$: the model's predicted probability for token $x_t$ given its context
• $N$: the total number of tokens in the corpus

The model is trained to maximize the log-probability of each correct next token given its context. Here, "correct" means "what appeared in the training corpus," not "what is factually true in the world."

This distinction is the root of the problem. The loss function contains no factual correctness term, no mechanism for cross-referencing claims across documents, and no reward for consistency with verified external sources. A factual error that appears consistently across thousands of web pages will be learned as a strong pattern exactly as a true fact appearing with the same frequency would be. The model encodes the statistical co-occurrence structure of tokens without any representation of the epistemic status of the claims those tokens compose.

There is also an amplification dynamic at play. The internet has a tendency to copy and redistribute content. A single article containing a factual error may be scraped, republished, summarized, and cited across dozens of derivative pages, each appearing as a separate training example. The model sees what looks like consensus (many sources agreeing) when in fact there is only duplication. This means the model may learn erroneous facts with higher confidence than the original single-source error would warrant, because the training signal artificially strengthens the incorrect claim through repetition.

Not all topics receive equal coverage on the internet. Widely documented phenomena (major historical events, famous figures, popular programming languages, capital cities) appear in billions of training tokens across diverse contexts. Obscure entities (small companies, local politicians, minor historical figures, niche scientific subfields) appear in tens or hundreds of tokens at best.

This creates a direct imbalance in the model's statistical knowledge. Well-represented entities have strong, consistent signal across many contexts, which allows the model to build a stable internal representation of the associated facts. Under-represented entities have weak or noisy signal, often from a small number of sources that may themselves be unreliable.

When asked about a tail entity, the model faces a harder inference problem: it cannot rely on many consistent exposures. Instead, it generalizes from surface-level patterns: "this looks like the type of question that should have a date/name/place answer." The model knows the form of the answer it should produce, and it knows the general vocabulary and discourse style associated with the topic domain, but the specific facts it generates are extrapolated from thin signal rather than reliably remembered.

The knowledge imbalance is compounded by the way coverage correlates with cultural and linguistic biases. English-language sources dominate most training corpora. Events important in non-English-speaking regions may be under-represented simply because less English-language content was produced about them. The model's knowledge is uneven across topics and systematically skewed by the demographics and cultural assumptions of the web content that gets indexed and crawled.

The internet frequently disagrees with itself. For any given factual question, you can find confident conflicting answers across different sources. Consider a claim like "Person X received Award Y in year Z." Multiple sources might report different years. Multiple sources might attribute the same quote to different people. Dates for historical events sometimes vary by source, particularly when primary sources are rare.

The model sees all of these during training and has no oracle to adjudicate between them. What does it learn? It learns a weighted average of the conflicting signals, where the weights are proportional to how often each version of the claim appears. The most-cited version wins, which may or may not be the correct one.

For very common facts, the correct version is typically cited far more than any incorrect alternative, so the weighted average converges to something accurate. For uncommon or contested facts, the distribution of citations may not favor accuracy. The model's "consensus" answer may simply reflect which version of the claim was most frequently published rather than which was most carefully verified.

The model has no concept of source reliability. A well-edited Wikipedia article, a peer-reviewed journal paper, a tabloid blog post, and a social media thread all contribute to the training signal with equal weight per token. There is no quality weighting in the standard pretraining objective. Some training pipelines apply quality filters that upweight curated sources, but these filters are imperfect and cannot eliminate the fundamental equalization that the loss function performs.

This matters most for facts that are close to true. A completely fabricated claim may simply not appear in enough training data to produce a strong signal. A partially wrong claim, however, may appear in many sources (because most sources got most of it right while disagreeing on one detail), generating a muddled but confident-sounding mixture of correct and incorrect information.

Pretraining on web data is usually followed by supervised finetuning (SFT) on instruction-following datasets. These datasets are created by human annotators who are asked to write ideal responses to a given prompt. Annotators are human, and humans make mistakes, disagree, and have knowledge gaps of their own. More subtly, annotators writing responses for topics outside their expertise may produce fluent, authoritative-sounding text that is factually wrong.

The model trained on SFT data learns the style of confident, well-structured prose, because that is what annotators tended to produce when demonstrating good responses. The formatting and rhetorical pattern of confident knowledge-sharing is reinforced even when the underlying claims are uncertain or incorrect. This is not the annotators' fault: they are doing their best with the knowledge they have. But the structural consequence is that the model learns to sound more certain than it should.

The SFT problem also interacts with a selection bias: annotators are more likely to write detailed, authoritative responses for topics they know well, and shorter, more hedged responses for topics where they are less confident. This means the model receives many examples of detailed, confident prose for well-known topics and fewer examples of careful hedging for under-known topics, reinforcing the pattern of overconfidence precisely where it is least warranted.

Beyond individual annotator errors, SFT datasets also suffer from coverage gaps that parallel the pretraining data gaps. Instruction-following datasets are created with particular use cases in mind: question answering, summarization, code completion, and similar tasks. They may have thin coverage of rare query types where hallucination is most likely. This means the model gets very little practice with exactly the queries it is most likely to fail on, while getting abundant practice with the queries it would handle correctly regardless.

There is also a subtler issue with the temporal consistency of SFT annotations. Annotation projects often span weeks or months, and different annotators may have different levels of up-to-date knowledge about the same topics. A question about an evolving scientific field or a recent product may receive different answers depending on which annotator handled it and when. These inconsistencies enter the SFT dataset and train the model on conflicting "correct" responses for the same type of query, which can degrade calibration and produce inconsistent outputs at inference time.

The path from raw web data to training-ready tokens involves many processing steps, and each introduces its own opportunities for error propagation. Understanding this pipeline helps clarify why "better data" is a difficult challenge.

Raw text from web crawls is typically filtered using heuristics: minimum document length, minimum token entropy, language identification, perplexity filtering against a small reference model, and blacklist-based removal of known low-quality domains. Each filter catches some noise but introduces its own biases. Perplexity filtering against a language model, for example, tends to remove text that is unusual or non-standard in surface form, which can over-remove domain-specific technical content that is perfectly accurate but simply uses different vocabulary patterns than the reference model learned.

Deduplication is applied to remove duplicate and near-duplicate documents. Exact deduplication removes documents that appear multiple times verbatim. Fuzzy deduplication removes documents that are highly similar. Both are necessary to prevent overweighting of content that was widely republished. But deduplication also changes the effective frequency of entities: if many different web pages discuss the same fact using slightly different wording, deduplication may collapse them into fewer examples, reducing the training signal for that fact below what the raw token count suggests.

The result is that the model's training data is the output of a complex pipeline with many implicit choices, each of which shapes which facts get encoded reliably and which do not. The relationship between "how many times did fact X appear in raw web crawl data" and "how strongly is fact X encoded in the model" is mediated by all of these pipeline steps, making it difficult to predict or audit from the outside.