Sentence Meaning and Natural Language Inference › The NLI Task and Its Datasets

The modern NLI task is usually presented as a supervised classification problem. Each example has a premise, a hypothesis, and a label. The model outputs a probability distribution over entailment, contradiction, and neutral.

The task is not symmetric. The premise is the evidence; the hypothesis is the claim. Swapping them changes the inference. This is why NLI models are trained on ordered pairs, not unordered sets of sentences. It is also why the evaluation has to respect the order when it reports per-direction results.

The Recognizing Textual Entailment challenges established an application-oriented benchmark in the mid-2000s. RTE-1 and RTE-2 used binary entailment versus non-entailment judgments on small, hand-labeled text-hypothesis datasets. Their goal was to capture inferences needed across NLP applications, rather than to isolate controlled templates in the later HANS style. The RTE-2 organizers' announcement states the applied inference task and its relation to those applications.

The Stanford Natural Language Inference dataset, SNLI, enabled a large-scale supervised phase of NLI research. Bowman and colleagues introduced it in their 2015 paper. SNLI contains about 570,000 human-written premise-hypothesis pairs before filtering out cases without a consensus label. The premises are image captions, and crowdworkers wrote hypotheses. SNLI uses the entailment, contradiction, and neutral labels used throughout this chapter.

MultiNLI, described by Williams and colleagues in their 2018 paper, expanded the task to multiple genres. It contains about 433,000 pairs across ten genres, including fiction, government documents, travel guides, and telephone conversations. MultiNLI has matched and mismatched development/test splits. Five genres occur in its supervised training split; all ten occur in development and test. Mismatched evaluation therefore tests transfer to genres absent from that supervised split, without establishing that a pretrained encoder has never encountered those genres.

XNLI, described by Conneau and colleagues in their 2018 paper, uses newly collected English premise-hypothesis pairs translated into 14 additional languages, for 15 languages including English. It is a standard benchmark for cross-lingual transfer. The translations introduce their own artifacts, and negation and word order can differ substantially across languages, so a model that does well on English MNLI is not guaranteed to do well on XNLI.

Later datasets targeted specific weaknesses:

• ANLI (Adversarial NLI) used a human-and-model-in-the-loop process. Annotators wrote examples that fooled current models, then models were retrained, then new examples were written. The resulting benchmark tests models against those adversarial examples.
• HANS (Heuristic Analysis for NLI Systems) constructs controlled cases where lexical-overlap, subsequence, and constituent heuristics succeed or fail. It is intended to diagnose those heuristics, not to represent the full distribution of inference in text.
• MedNLI tests clinical inference grounded in patient histories. SciTail derives hypotheses from school-science questions and answer choices, with retrieved web sentences as premises.
• DocNLI uses document-level premises and variable-length hypotheses, extending the input beyond isolated sentence pairs.

These datasets do not all share the same label alphabet. Binary non-entailment, for example, does not preserve a separate contradiction category:

Different datasets address scale, genre coverage, language transfer, or targeted failure diagnosis. Select evaluations for the capabilities needed in the application rather than treating their scores as interchangeable.

NLI datasets are not neutral reflections of semantic competence. They contain artifacts from how they were constructed.

A useful artifact check is the hypothesis-only baseline. If you train a classifier on the hypothesis alone, ignoring the premise, you can achieve accuracy well above chance on SNLI and other datasets. The hypotheses were written by crowdworkers who tended to use certain words and constructions for certain labels. For example, hypotheses containing "not" or "nobody" may be more likely to be contradictions. A model can learn these label-associated patterns without learning anything about the premise.

This does not mean SNLI is useless. It means that high accuracy on SNLI is not proof of semantic understanding. It also means that evaluation must include controls for hypothesis-only information, lexical overlap, and other shortcuts. The right response is not to discard the dataset but to interpret its scores with appropriate skepticism.

Which label is hardest depends on the model and evaluation set. Neutral includes varied cases, so inspect per-class errors rather than assuming a fixed difficulty order. Predicting neutral more often can raise its recall while increasing false-neutral errors; predicting it on every example would give perfect neutral recall when gold neutral examples are present, without establishing any inference ability. Model uncertainty is separate from the semantic neutral relation and may call for abstention instead.

Class imbalance varies by dataset and split. Always check the label distribution before reporting accuracy. A model that predicts the majority label can look better than it is. Macro-F1 gives each label equal weight in the aggregate score. Per-class recall reports a separate value for each label, so minority class recall remains visible.