Lexical Semantics and Word Senses › Limitations and Impact

The first limitation is conceptual. Word senses are not natural kinds with crisp boundaries. They are lexicographic constructs designed to serve dictionary users, translators, and computational systems. Different inventories draw the boundaries differently, and even trained annotators disagree. A model's "error" may sometimes reflect a defensible alternative analysis, but annotator disagreement does not itself impose an upper bound below 100% on agreement with a fixed adjudicated gold set. When you report WSD accuracy, you are reporting agreement with a particular inventory and annotation guideline, not agreement with semantic truth. This distinction is important for interpreting results. A model that disagrees with the gold label may still have made a defensible semantic choice. A model that agrees with the gold label may have chosen it for the wrong reason. Evaluation measures agreement, not understanding.

The second limitation is practical. An explicit WSD module adds decisions about the inventory, integration, and evaluation. New or maintained sense annotation may also be needed, depending on the method and the available resources. Its value should be tested in the intended application. A translation model can choose a contextual translation without emitting a WordNet label; a classifier can predict sentiment directly from a phrase. Explicit labels remain useful when an application needs inspectable lexical decisions or links to a lexical resource. Benchmark improvements and downstream improvements are separate measurements, so a historical benchmark score alone cannot decide which architecture to use.

The third limitation is evaluation. Public WSD benchmark examples may overlap LLM pretraining data. Option order and wording can affect prompted WSD results. A candidate-choice prompt and free-form generation have different output interfaces. Report the label matching and scoring rules for the interface you use. If the task requires an inventory identifier, constrain decoding or validate the returned identifier. Report the inventory and your contamination checks. Choose a baseline that tests the claimed improvement. You should also consider human evaluation for cases where the inventory is ambiguous or the model's explanation is plausible but the label is wrong. Evaluation is not a single number. It is a protocol with assumptions. The assumptions should be stated and checked.

The fourth limitation is coverage. A fixed inventory may lack an entry or a distinction required in a target domain. Check the selected release against actual vocabulary and usages; English Princeton WordNet coverage does not establish the coverage of a multilingual derivative. Target-domain evaluation can reveal changes in context, sense priors, or inventory coverage. Adapt the model, priors, data, or inventory when those checks show a need. In multilingual settings, word alignment and label mappings can also introduce errors. A large entry count alone does not show that the resource represents the distinctions an application needs.

Three applications show why a sense distinction can matter. A translation system must choose an appropriate target-language expression for the context; the bank examples contrast banque and rive. A retrieval system can use typed lexical relations for query expansion, with precision/recall effects measured on its queries. A sentiment resource can associate different scores with different senses, as SentiWordNet does at synset level. These uses do not all require a separate WSD module. Evaluate the output the application needs, whether it is a translation, a ranked document list, a sentiment decision, or an explicit sense identifier.

The following chapters examine sentence meaning, semantic roles, relation extraction, and entity linking. Keep each task's output contract explicit. This chapter assigned inventory senses; the later tasks introduce their own targets and output definitions.

For practitioners, a few guidelines are worth keeping:

• Define the inventory before you build the system. A WSD result is meaningless without a clear label space and version.
• Report a training-data MFS baseline or clearly label a first-WordNet-sense proxy. Add other baselines that test the particular claimed improvement.
• Report results by part of speech and frequency band when those groups matter to the task. Aggregate accuracy can hide poor performance on rare senses.
• Consider coarse-grained evaluation when fine distinctions are unstable, disclose the merged distinctions, and measure reliability and application usefulness.
• Control for contamination and prompt sensitivity when using LLMs. A public benchmark score is not enough.
• Consider whether explicit WSD is necessary. Sometimes a sense-aware representation helps; sometimes an end-to-end model is better.

These guidelines are not a checklist for every project. They are a set of questions to ask before making claims. The most important question is the first one: what is the inventory? Without a clear answer, the rest of the evaluation is unreliable. The second most important question is whether the system beats the baseline. If it does not, investigate the tradeoff and test whether a different benefit, such as rare-sense accuracy, justifies the complexity. The third is whether the evaluation measures what the application needs. A system that performs well on a general benchmark may fail on a specialized domain. A system that performs well on common senses may fail on rare senses. A system that performs well on a public benchmark may fail on a new test set. These questions help connect lexical semantics to practical language AI.