Lexical Semantics and Word Senses › Word-Sense Disambiguation

The decomposition also explains a frequency baseline. When every candidate has the same positive context likelihood, choosing a largest prior is the MAP decision. The actual accuracy of an MFS baseline is the fraction of test targets carrying the senses selected from the stated training data; it is not a fixed percentage. Report it on the same targets, parts of speech, and inventory as the evaluated system. A context-sensitive model can have lower aggregate accuracy yet help on a costly rare sense, so baseline comparisons should also reflect the application's error costs and sense-frequency groups.

WSD comes in several task variants:

• Lexical sample: The evaluation focuses on a fixed set of target words. The protocol specifies whether sense-tagged training data is supplied; a supervised system may use word-specific classifiers.
• All-words: Disambiguate the eligible annotated targets throughout a text, rather than a predefined lexical sample. The protocol determines the target parts of speech, inventory coverage, and any excluded uses.
• Domain-specific: The data, the sense inventory, or both focus on a domain such as biomedicine or law. A domain-text evaluation can still use a general-purpose inventory.
• Multilingual: The system disambiguates words across languages, for example using parallel corpora or multilingual encoders.
• Coarse-grained: The inventory merges fine distinctions. This reduces the label distinctions required, but annotation reliability and system performance still need measurement.

Lexical sample tasks are useful for controlled experiments. They allow researchers to focus on a small set of words with enough training data to support supervised learning. All-words evaluation samples eligible targets across documents and can expose words or senses absent from a lexical-sample training set. Whether it matches an application depends on the documents and target protocol. A system that performs well on lexical sample may struggle on all-words because it has not learned to handle the long tail. Domain-specific tasks are important because sense distributions vary by domain. A multilingual task can test cross-language transfer by training in one language and evaluating in another. Coarse-grained tasks are useful when fine-grained distinctions are unstable. Each variant answers a different question, and results from one variant should not be generalized to another without care.

WSD is hard for several reasons, and they interact.

Sense-tagged corpora can have strongly imbalanced word and sense counts. MetricWSD (Chen et al., 2021), for example, studies both kinds of imbalance in SemCor training data. Such skew can leave little supervision for rare senses. The observed distribution depends on genre, domain, sampling, and annotation policy; it is not a law that every corpus must have the same pattern. Measure the frequency distribution in the data used for the experiment.

Context is sparse. The words that disambiguate a sense may not appear in a short window. In "The bank was steep," steep can suggest a riverbank, but it can also describe another slope. "She walked toward the bank" permits a financial-institution reading or a river-edge reading until more context is supplied. Duan and Yates (2010) motivate their WSD approach with the need for world knowledge. Disambiguating cues can also lie beyond the local window. A sentence may mention a bank early and reveal that it is a riverbank much later. Leacock, Chodorow, and Miller (1998) found that adding topical features to local context helped on some of their evaluated words, but not all. A model that only looks at a small window may miss the evidence.

Some sense distinctions are easier to apply than others. The financial-versus-river distinction for bank differs from separating two closely related financial uses. Check whether the annotation guidelines let annotators apply the intended distinction consistently. A model can fit noisy labels, so training accuracy alone does not establish that it has learned the intended semantic distinction. This is a reason to inspect disputed cases and held-out errors, rather than assume every finer label is useful supervision.

Sense annotation requires guidelines, annotator preparation, and quality control; professional lexicographers are not a universal prerequisite. SemCor pairs Brown Corpus passages with WordNet senses. The processed SemCor training set used by Blevins and Zettlemoyer (2020) contains 226,036 annotated examples. That count describes their preprocessing and target selection, rather than every release or possible annotation policy. When planning a new corpus, budget the annotation work for the target distinctions and measure agreement on trial examples.

Domain changes can reduce a WSD model's accuracy. A word's dominant sense in newswire may differ from its dominant sense in clinical notes or social media. A model trained on one domain may confidently predict the wrong sense in another. Changes in sense priors can contribute to transfer errors. A sense that is rare in general English may be common in a specialized domain. A model that relies on the general prior may systematically choose the wrong sense. For domain adaptation, measure how sense distributions and errors change. Depending on the inventory and data, fine-tuning, prior adjustment, or another intervention may be appropriate.

The inventory may not fit the data. A word use may be metaphorical, metonymic, or novel. It may fall between two senses or combine them. A forced-choice WSD system must still pick a label, which can produce misleading evaluations. If the inventory lacks a sense for a common use, the model is forced to choose an available label. If annotators assign a fallback label from the inventory, a semantically reasonable prediction can still disagree with that label. If the protocol excludes the use from scoring, report it separately as a gap in inventory coverage. If the inventory splits a single use into two senses, the model may be penalized for choosing the other split. The inventory and annotation guidelines can leave a use uncovered or its reference label ambiguous. For cases that enter scoring, the model’s prediction and the accepted reference labels determine whether a disagreement is counted.

A frequency-only classifier can score highly when its training-selected senses also dominate the evaluated targets. Its accuracy is the fraction of those targets carrying those labels. The BEM comparison below gives one measured example; it does not establish that improvements over MFS are generally small. Compare both overall and rare-sense outcomes on the stated inventory and target set.

WSD evaluation uses several metrics and protocols.

For a fixed set of (N>0) annotated targets, submit at most one response per target. Let (A) be the number attempted and (C) the number assigned a correct sense, with (0\le C\le A\le N).

Accuracy when all targets are attempted is (C/N). Under this fixed-target abstention protocol, precision is (C/A) for (A>0), and recall is (C/N).

When (C>0), their harmonic mean simplifies to (F1=2C/(A+N)). We extend this count formula to (C=0), giving F1 of zero.

With no attempts, precision remains undefined while recall and this F1 convention give zero. If every target is attempted, precision, recall, F1, and accuracy therefore coincide, including the all-wrong case.

With the other predictions and target count fixed, abstaining on an incorrect answer leaves recall unchanged; abstaining on a correct answer lowers recall. Recompute precision from the remaining attempted answers; it is undefined if no attempts remain.

The gold protocol supplies the targets; this is not a free decision to label arbitrary words. Disclose zero-case conventions and how multiple acceptable labels are handled, and report per-part-of-speech results when they matter.

Include a most-frequent-sense comparison when evaluating whether context improves over training-set frequency. Compute each lemma's baseline from the specified training data, state the fallback for unseen lemmas, and use the same gold targets as the model. If the implementation instead uses the first WordNet sense, identify it as that proxy. A lower aggregate score warrants investigation, but usefulness also depends on rare senses, error costs, coverage, and the downstream task.

Standard datasets include:

• Senseval: Early evaluations with edition-specific protocols. Senseval-1 was a lexical-sample exercise; Senseval-2 included both lexical-sample and all-words tasks. The BEM evaluation discussed below uses the later Senseval-2 and Senseval-3 English all-words datasets.
• SemEval-2007 Task 17: English lexical-sample, semantic-role-labeling, and all-words subtasks.
• SemEval-2013 Task 12: Multilingual noun sense disambiguation using BabelNet 1.1.1 or its WordNet/Wikipedia projections. The annotated targets included eligible multiword expressions and named entities.
• SemEval-2015 Task 13: Multilingual all-words word-sense and entity disambiguation using BabelNet 2.5.1. Unlike the fixed-target example above, participating systems were not given the fragments to disambiguate. When counting true positives, the scorer gave equal weight to multiple answers submitted by a system for one instance. It ignored predicted fragments absent from the gold standard.
• SemCor: A sense-tagged subset of the Brown Corpus, widely used for WSD training. It has also served as an evaluation corpus in published WSD studies.
• OntoNotes: A multilingual corpus whose documented English inventory groups fine-grained WordNet senses.
• Word-in-Context (WiC): A binary task that asks whether a word has the same sense in two sentences. It avoids fine-grained sense labels but still tests contextual sense discrimination.

WiC deserves special attention. It reformulates WSD as a comparison problem: given two sentences containing the same target word, decide whether the word is used in the same sense. The model outputs a binary judgment rather than a sense ID. The original WiC dataset nevertheless used WordNet, VerbNet, and Wiktionary examples and pruned subtle distinctions using resource relations. One way to implement this binary comparison is to compare contextual representations of the two target occurrences. However, WiC does not tell you which sense is used, only whether two uses match. It is a complementary evaluation, not a replacement for inventory-based WSD. A model that excels at WiC may still fail at assigning a specific sense label. A model that assigns labels may still fail at comparing two uses. With one adjudicated label per occurrence in a shared, aligned inventory, comparing two correct WSD labels settles a pair task defined by label equality. If the protocol accepts several labels per occurrence, specify how the pair task compares the accepted sets. For example, with accepted sets ({a,b}) and ({b,c}), predictions (a) and (c) are both accepted even though they differ and the sets share (b). A binary judgment alone leaves the sense identity undetermined. Across different inventories or granularities, success on one task does not guarantee success on the other.

Benchmark contamination is a serious concern for modern LLMs. WiC and SemEval examples may appear in pretraining data. A model that scores well on WiC may have memorized the benchmark rather than learned to discriminate senses. Check whether the evaluation data was filtered from pretraining when training data or documentation allow it. If you cannot verify filtering, say so. Prefer hidden or newly constructed test sets when making strong claims. Contamination is especially difficult to detect because LLMs are trained on massive web corpora. A public benchmark example may appear in a blog post, a paper, a tutorial, or a code repository. Even if the exact example is not present, paraphrases or discussions of the example may be present. A high score on a contaminated benchmark alone does not establish generalization to unseen sense-disambiguation contexts.