Sentence Meaning and Natural Language Inference › Robust NLI Evaluation

NLI evaluation can test controlled semantic distinctions alongside transfer across genres, domains, and languages. Agreement with one benchmark's labels answers a narrower question than success across all those settings.

In-distribution test sets share the same annotation process and genre distribution as training data. They are useful, but they can overestimate generalization. A model can score very high in-distribution while relying on cues that vanish the moment the data changes.

Out-of-distribution evaluation uses a different genre, domain, or annotation process. MultiNLI's mismatched split is an early example. Domain-specific datasets like MedNLI test whether a model can handle medical text. Cross-lingual evaluation with XNLI tests transfer across languages.

Adversarial evaluation is more targeted. ANLI uses human-written examples designed to fool models. HANS uses controlled templates to test specific heuristics. Contrast sets make small, meaningful perturbations to examples, typically changing the gold label. These sets are not necessarily representative of natural text; they test particular local distinctions. Failure on a controlled case can identify a distinction to investigate even when aggregate accuracy is high.

The hypothesis-only baseline is the simplest artifact check. Train a model on the hypothesis alone. If it performs well above chance, the dataset contains label-associated patterns in the hypotheses. This does not invalidate the dataset, but it means that premise-aware models may be learning some of the same shortcuts.

Lexical overlap is another common shortcut. A model may predict entailment whenever the hypothesis words appear in the premise. HANS tests this with examples like:

• Premise: "The lawyer questioned the witness."
• Hypothesis: "The witness questioned the lawyer."

The words overlap heavily, but the relation is neutral. A model that relies on overlap may predict entailment. A model that represents argument order and thematic roles should predict neutral.

HANS distinguishes three heuristics:

• Lexical overlap: if all hypothesis words appear in the premise, predict entailment.
• Subsequence: if the complete hypothesis is a contiguous subsequence of the premise, predict entailment. This is the definition used in HANS Table 1, rather than the broader mathematical use of subsequence.
• Constituent: if the complete hypothesis corresponds to a syntactic constituent of the premise, predict entailment.

A heuristic can agree with the gold labels on many examples without being a valid inference rule. Cases where the heuristic and gold label diverge test whether that shortcut is sufficient. Passing those cases still requires follow-up checks; a model may use another shortcut.

A good NLI evaluation battery covers the phenomena that make sentence meaning hard. The following categories are a useful starting point.

• Negation: "The door is open" versus "The door is not open." Test whether predictions respond to its scope rather than only its presence.
• Numerals and quantities: "At least three dogs" versus "Exactly three dogs." "More than five" versus "Fewer than five." Numerical inference is compositional and easy to perturb.
• Entity swapping: "Alice thanked Bob" versus "Bob thanked Alice." High lexical overlap, different roles.
• Quantifier scope: "Every student read a book" versus "There is a book that every student read." The first does not entail the second under the distributive reading.
• Temporal relations: "Alice left before Bob arrived" versus "Bob arrived before Alice left." Order matters.
• World knowledge: "The Eiffel Tower is in Paris" versus "The Eiffel Tower is in France." The inference requires geographical knowledge.
• Presupposition: "John stopped smoking" versus "John used to smoke." On the intended presuppositional reading, prior smoking remains backgrounded under negation; test that reading explicitly.
• Lexical ambiguity: "The bank raised rates" versus "The river bank is steep." Same word, different senses.
• Collective versus distributive: on the collective reading, "Alice and Bob lifted the piano" does not entail "Alice lifted the piano alone."
• Long premises: A relevant sentence buried in a long document. Test whether adding irrelevant context changes the relation prediction.

These categories overlap: one example can involve negation, scope, and world knowledge. Report category-level results alongside the aggregate score so that a strong category does not conceal a failure on an important one. Deployment decisions also need evidence from the intended application.

For models trained on public web text, overlap with publicly released evaluation examples is a possibility that requires investigation. A benchmark score alone cannot establish or rule out memorization. The GPT-3 study examined training/test overlap; whether a particular NLI dataset contaminated a particular model still requires evidence about those inputs.

Contamination is hard to prove or rule out. Practical mitigations include:

• Evaluate on a private or recently collected dataset.
• Check n-gram overlap between the evaluation set and known pretraining corpora, if available.
• Use time-split evaluation, where the test set was created after the model's training cutoff.
• Report multiple benchmarks, including adversarial and domain-specific sets.
• Treat unusually high zero-shot scores with suspicion.

Prompt sensitivity also matters for LLM-based NLI. Test whether predictions are stable under prompt changes rather than relying on one phrasing. An evaluation can:

• Test multiple prompt templates.
• Permute the order of label options.
• Report variance across prompts.
• Use constrained scoring over label tokens rather than free generation when possible.
• Report decoding settings, including temperature, top-p, and random seeds.

Even greedy decoding is not a guarantee of determinism across hardware and batch sizes. If reproducibility matters, fix the environment and report it. A reader who cannot reproduce your setup cannot check your conclusions.

The following measurements answer different evaluation questions:

• Accuracy: overall correctness.
• Macro-F1: averages per-class F1, giving each label equal weight.
• Per-class precision and recall: report all labels; the weakest class depends on the model and evaluation set.
• Calibration: whether predicted probabilities match empirical correctness. A model that is 90% accurate but always outputs 99% confidence is badly calibrated.
• Selective accuracy: accuracy on examples where the model's confidence exceeds a threshold. This is useful when a system can defer to a human.
• Adversarial accuracy: performance on HANS, ANLI, or domain-specific stress tests.
• Confidence intervals: resample independent evaluation units and recompute the metric. Cluster related pairs by source document or template when needed. State the population and resampling scheme; an interval conditional on a fitted model does not include all training, annotation, or distribution-shift uncertainty.
• Paired comparisons: for accuracy, McNemar's test uses paired correct/incorrect outcomes. For macro-F1 or another aggregate metric, use paired resampling or randomization appropriate to that metric and sampling design, applying the same sampled units to both models.
• Human agreement: annotator agreement on the evaluation set, especially for neutral versus entailment.

A small test set can produce large confidence intervals. Whether a difference of one or two examples is informative depends on the sample size, pairing, and uncertainty; the count alone is insufficient. An interval describes uncertainty under its stated sampling assumptions. It cannot make a hand-picked diagnostic battery representative. Test choice also depends on the metric and data dependence.

If you are building or deploying an NLI system, a practical protocol looks like this:

1. Define the label semantics for your domain. Is a presupposition an entailment? Is a plausible inference enough, or do you need strict logical consequence?
2. Check the label distribution in your training and test data. Report majority-class baselines.
3. Run a hypothesis-only baseline to detect annotation artifacts.
4. Build a diagnostic battery covering negation, numerals, entity swaps, quantifiers, temporal relations, and presupposition.
5. Test lexical overlap shortcuts with controlled pairs.
6. Evaluate out-of-distribution on a different genre or domain.
7. Check for contamination if using a public benchmark or an LLM.
8. Test prompt sensitivity if using an LLM.
9. Report per-category results and confidence intervals.
10. Do an error analysis on a random sample and on the worst-performing category.

This protocol will not make NLI solved. It will make your claims more honest.