Sentence Meaning and Natural Language Inference › How NLI Models Work

NLI models have evolved from feature-based classifiers to large pretrained transformers. The core problem remains the same: represent the relationship between two sentences.

Earlier RTE systems often combined hand-engineered features. For example, MENT's RTE-2 system used syntactic and thesaurus features with an alignment score. A feature-based baseline can combine:

• Lexical overlap between premise and hypothesis.
• Word alignment and edit distance.
• Syntactic parse features.
• Named entity overlap.
• Negation detection.
• A text-similarity feature, building on TF-IDF or BM25. Such a score measures a form of similarity or relevance, not entailment by itself.

A logistic regression or SVM classifier could combine these features. Their limitations depend on the chosen features: a word-count representation omits order, while explicit syntax or negation features can preserve some distinctions. A classifier that succeeds through lexical overlap still needs tests where overlap and entailment disagree.

Two common designs illustrate different ways to represent the sentence pair.

A bi-encoder encodes the premise and hypothesis separately:

$$u = f_\theta(p), \qquad v = f_\theta(h)$$

where:

• $p$: the premise sentence
• $h$: the hypothesis sentence
• $f_\theta$: the encoder function parameterized by weights $\theta$
• $u$: the encoded representation (vector) of the premise
• $v$: the encoded representation (vector) of the hypothesis

It then combines the two vectors, for example:

$$z = [u; v; |u-v|; u \odot v]$$

Here $u,v \in \mathbb{R}^{d}$ have the same dimension. Semicolons concatenate vectors, $|u-v|$ takes the componentwise absolute difference, and $\odot$ multiplies corresponding components. The pair vector $z \in \mathbb{R}^{4d}$ has four blocks of $d$ entries. Its ordered $u$ and $v$ blocks can preserve direction even though the difference and product blocks are symmetric.

A classifier uses the combined vector $z$. Separate encoding allows sentence vectors to be cached and reused across pairs. It omits direct cross-sentence token interaction during encoding. A sentence encoder can still represent internal negation and role information, but the pair classifier must recover the relation from the fixed vectors.

A cross-encoder concatenates the premise and hypothesis into a single input:

$$[\text{CLS}] \ p \ [\text{SEP}] \ h \ [\text{SEP}]$$

where:

• $[\text{CLS}]$: the special classification token whose final hidden state is used for prediction
• $p$: the token sequence of the premise
• $[\text{SEP}]$: the special separator token marking segment boundaries
• $h$: the token sequence of the hypothesis

A transformer encodes the whole sequence. With an unblocked bidirectional attention mask, self-attention permits each premise token to attend to each hypothesis token. The final hidden state at the [CLS] position, or a pooled representation, is passed to a linear classification head:

$$p_\theta(y \mid p,h) = \operatorname{softmax}(W h_{[\text{CLS}]} + b)_y$$

where:

• $p_\theta(y \mid p,h)$: the predicted probability of label $y$ given premise $p$ and hypothesis $h$
• $y$: a label in {entailment, contradiction, neutral}
• $h_{[\text{CLS}]}$: the final hidden state at the [CLS] position
• $W$: the weight matrix of the linear classification head
• $b$: the bias vector of the classification head
• $\operatorname{softmax}(\cdot)_y$: the softmax function evaluated at the entry for label $y$

With a pooled hidden state $h_{[\mathrm{CLS}]} \in \mathbb{R}^{d}$, a three-label head uses $W \in \mathbb{R}^{3 \times d}$ and $b \in \mathbb{R}^{3}$. Its output has one logit per label. This hidden-state symbol is distinct from the hypothesis sentence $h$.

A cross-encoder requires joint encoding of each pair, whereas a bi-encoder can reuse an already computed sentence vector. Joint encoding permits token-level interaction across the pair; the resulting accuracy depends on the model, training data, and evaluation set. For a workload comparing many new pairs, this joint computation cannot use a cache of independently encoded sentences in the same way. Compute comparisons still depend on sequence length, architecture, batching, and hardware.

The special tokens [CLS] and [SEP] were introduced in Special Tokens. Their role here is structural: [CLS] aggregates the pair representation, and [SEP] marks the boundary between premise and hypothesis.

For this supervised classifier, take a nonempty dataset of $N>0$ labeled pairs with one gold label per pair. Minimize the mean cross-entropy loss:

$$\mathcal{L}(\theta) = -\frac{1}{N} \sum_{i=1}^{N} \log p_\theta(y_i \mid p_i, h_i)$$

where:

• $\mathcal{L}(\theta)$: the cross-entropy loss for model parameters $\theta$
• $N$: the number of labeled pairs in the dataset
• $i$: an index over training pairs
• $p_i$: the premise of the $i$-th pair
• $h_i$: the hypothesis of the $i$-th pair
• $y_i$: the gold label of the $i$-th pair
• $p_\theta(y_i \mid p_i, h_i)$: the model's predicted probability of the gold label
• $\log$: the natural logarithm

Minimizing this objective rewards higher probability on the supplied gold labels. One training recipe is to fine-tune a pretrained transformer on MNLI, then adapt it to a domain-specific dataset. This is a recipe to evaluate, not a guarantee of transfer.

Class weighting, label smoothing, and adversarial training are optional refinements. Their suitability needs separate evaluation; changing training weights, targets, or examples does not by itself establish semantic generalization. The objective teaches the model to match labels, and if the labels are predictable from shortcuts, the model will happily learn the shortcuts.

Large language models can perform NLI zero-shot or few-shot. A typical prompt asks:

Premise: ...

Hypothesis: ...

Does the premise entail the hypothesis, contradict it, or neither? Answer with one of: entailment, contradiction, neutral.

There are two ways to score the answer:

1. Generate the label as text and parse it.
2. Score the probability of each label token or verbalizer and choose the highest.

Free generation introduces decoding and parsing choices. Temperature zero makes token selection greedy, but does not by itself guarantee reproducible inference across hardware, software, or batching configurations. Prompt wording, demonstration choice, and demonstration order can also change predictions. Scoring label tokens fixes the candidate outputs, but still depends on the verbalizer and tokenization. For example, "entailment" may tokenize into multiple pieces, so score the sequence, not just one token.

Prompting an LLM provides a way to attempt the task without task-specific training examples. It does not establish an advantage over a fine-tuned cross-encoder. Compare model families on the same evaluation protocol and inspect their errors.