Sentence Meaning and Natural Language Inference › Limitations & Impact

NLI evaluates a relation under a specified labeling policy; it is not a complete theory of meaning. The three-label format does not identify every meaning layer. A policy may count a projected presupposition as entailment, or reserve entailment for asserted content. It must also state how to treat conversational implicature. Neutral is a residual category that mixes underspecification, missing world knowledge, ambiguity, and pragmatic effects. A model can achieve high aggregate accuracy while systematically failing on negation, quantifier scope, or entity swapping.

The datasets themselves carry artifacts. SNLI and MultiNLI were built with crowdworkers, and crowdworkers are sensitive to the hypotheses they are writing. Hypothesis-only baselines perform well above chance. Lexical overlap is a strong cue. HANS targets specified syntactic heuristics, while ANLI collects examples adversarially against models. A high MNLI score is evidence that a model has learned something useful, but it is not evidence that the model has solved sentence meaning.

Large language models add new evaluation problems. Public NLI examples may occur in pretraining corpora; establishing contamination for a particular model requires evidence about that model and its data. Prompt wording and demonstration order can change predictions. Greedy decoding does not by itself guarantee identical results across hardware or batching configurations. A responsible evaluation reports multiple prompts, checks for contamination, and includes adversarial and domain-specific tests. Calibration and selective accuracy measure probability reliability and accuracy at a stated coverage; neither establishes that the model represents its own lack of knowledge.

NLI models are also used as zero-shot classifiers. The source-consistency, evidence-reranking, and factual-consistency methods introduced earlier provide other concrete applications. They can help decide whether a retrieved passage supports a generated claim, whether a summary is entailed by a source document, or whether two product descriptions contradict each other. But they should not be treated as truth arbiters. The premise is assumed true. The model checks the relation, not the facts. A system that forgets this distinction can confidently endorse a premise that is itself false, or reject a hypothesis that happens to be true but is not supported by the given text.

The most important lesson is methodological. Evaluate NLI systems on the phenomena you care about, not just on a leaderboard. Build controlled pairs. Test negation and scope. Test presupposition. Test lexical overlap. Test domain shift. Report per-category results and uncertainty. Use human review for high-stakes decisions. The next chapters extend this semantic toolkit to semantic roles, event extraction, entity linking, and semantic parsing. Those tasks will reuse the same core ideas: composition, scope, lexical entailment, and the difference between what a sentence says and what a speaker means.