Embedding Evaluation › Limitations and Practical Guidance

Embedding evaluation is a field still working out its own foundations. The measures we use are imperfect proxies, and interpreting them requires care.

The biggest limitation is that no single evaluation captures all aspects of embedding quality. Word similarity tests semantic nearness but not syntactic properties. Analogy accuracy measures structured semantic relationships but is sensitive to the specific analogy categories included. Downstream evaluation is most reliable but depends heavily on which downstream task you choose and how much training data is available. Bias evaluation using WEAT detects specific pre-defined biases but cannot assess all forms of representational harm.

Coverage is a neglected dimension. Two embedding sets might have similar intrinsic scores while differing greatly in vocabulary size. The smaller vocabulary might score higher simply because its words are more common and easier to represent well. Reporting coverage alongside quality metrics is essential for fair comparisons, especially when comparing embeddings trained on different corpora or with different vocabulary construction strategies.

Visualization is both the most informative and most misleading evaluation tool. t-SNE and UMAP projections are excellent for qualitative exploration and debugging, but they should never be used as quantitative evidence. Cluster separability in a 2D projection depends heavily on projection parameters (especially t-SNE's perplexity), the random initialization, and which words you choose to include. It's easy to make embeddings look impressive or poor depending on how you configure the visualization. Any visualization should clearly report its parameters so readers can assess what they're seeing.

Bias evaluation deserves to be a standard part of any embedding assessment, especially for embeddings used in user-facing systems. WEAT provides a starting point, but it tests specific pre-defined associations. Real-world bias is multidimensional and often implicit, operating through indirect associations that WEAT doesn't test. Running WEAT and finding small effect sizes does not mean the embeddings are bias-free; it means they don't exhibit the specific biases you explicitly tested for.

The field has also begun questioning whether static word embeddings should be evaluated this way at all. Static embeddings give each word one vector regardless of context, which is fundamentally at odds with the context-dependence of meaning. Evaluation on word similarity pairs, which present words out of context, may be measuring something that doesn't quite exist: a word's context-free meaning. This is one of the motivations for contextualized representations, which we'll explore in the next part.