Attribution and Citation: Sourcing LLM Outputs › Limitations and Practical Challenges

Attribution and citation evaluation face a set of persistent technical and conceptual challenges that current systems have not fully solved.

NLI models are imperfect judges. The automatic evaluation of whether a passage supports a claim depends on NLI models, and these models have their own error rates. They may fail on claims that require background knowledge not present in the passage, on claims that involve negation (a passage that says "X does NOT cause Y" can be misclassified as supporting "X causes Y"), or on claims that are correct by common knowledge but go beyond what any single passage states. NLI models trained on standard entailment datasets may also be poorly calibrated for the specific types of claims that appear in specialized domains like medicine, law, or finance.

Multi-hop attribution is hard. Some claims require combining information from multiple passages to verify. "Compound A was synthesized by the team that also discovered Compound B" requires knowing who synthesized A and who discovered B from potentially different passages. Current NLI-based evaluation treats each passage-claim pair independently and cannot handle this kind of multi-hop reasoning. Systems that aggregate evidence across passages are more capable but much harder to implement and evaluate.

Granularity mismatch. Attribution evaluation works best when claims are simple, atomic statements. Real model outputs often contain complex, multi-part sentences that mix claims from different sources. Automatically segmenting a generated paragraph into evaluable claims is a non-trivial task that introduces its own errors. Sentence boundaries do not always correspond to claim boundaries: a single sentence can contain multiple independent factual claims, and a claim can sometimes span multiple sentences.

Coverage vs. accuracy. Models can improve their measured attribution metrics by being conservative: cite fewer things, and the things you do cite are more likely to be accurate. But a highly conservative system that cites nothing achieves perfect precision while providing zero value. Evaluation frameworks must penalize both over-citation and under-citation, which requires defining what "adequate citation coverage" looks like for a given query, a question that does not have a universal answer.

Source quality is out of scope for attribution. Attribution evaluation checks whether a claim matches its cited source; it does not check whether the source is correct. A perfectly attributed response that cites a paper with fabricated data will score well on attribution metrics while misleading the reader. This is why source curation and corpus quality are architectural concerns, not just evaluation concerns.

Temporal decay. Even a correctly attributed claim may become outdated. A source from 2019 may have been superseded by 2024 research. Attribution evaluation that ignores the temporal dimension of sources will undervalue the importance of citation freshness. In fast-moving fields, a well-attributed but stale response can be just as misleading as a hallucinated one. Systems that track citation dates and flag outdated sources add a valuable additional layer to attribution quality.

The annotation agreement problem. Human evaluation of attribution quality suffers from lower inter-annotator agreement than might be expected. Judgments about whether a passage "supports" a claim involve interpretive choices: how strong does the evidence need to be? Does a passage that is consistent with a claim without directly stating it count as support? Different annotators apply different standards, which makes it difficult to create gold-standard evaluation datasets with high confidence that the labels are correct. This ambiguity flows through to the NLI models trained on these labels, which inherit the annotation disagreements as model uncertainty.

Despite these limitations, attribution evaluation is substantially more informative than accuracy evaluation alone. A system that is accurate but uncited is just as unverifiable as a system that hallucinates, from the user's perspective. Attribution provides the transparency layer that separates a trustworthy AI assistant from an oracle that demands blind faith. The imperfections of current attribution methods do not undermine the importance of the goal; they motivate continued work on better evaluation frameworks, better training methods, and better source curation strategies.