Embedding Evaluation › Embedding Visualization
Language AI HandbookAll chapters Embedding Evaluation › Embedding VisualizationOne section · Sections in this chapter5 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Numbers are abstract. Visualizations make semantic structure tangible. By projecting high-dimensional embeddings to 2D, you can visually inspect whether related words cluster together and unrelated words sit apart. Visualization is most useful during development for debugging and hypothesis generation: if you notice that your animal words don't cluster as expected, you can investigate why.
t-SNE: Local Structure Preservation
t-Distributed Stochastic Neighbor Embedding (t-SNE) is the most popular dimensionality reduction method for visualizing embeddings. It works by defining a probability distribution over pairs of high-dimensional points, where nearby points have high probability of being "neighbors," and then optimizing a low-dimensional layout to match that distribution as closely as possible.
The key insight of t-SNE is what makes it different from PCA or MDS. In the high-dimensional space, t-SNE uses a Gaussian distribution to define neighborhood probabilities. In the low-dimensional projection space, it uses a Student's t-distribution with one degree of freedom. The t-distribution has heavier tails than the Gaussian. This asymmetry is intentional: it resolves the "crowding problem" that plagues other dimensionality reduction methods. In high dimensions, there's lots of room for points to be moderately close to each other. When you project to 2D, all those moderately-close points would have to collapse together, making clusters indistinguishable. The heavy-tailed t-distribution in the projection space means that moderately-close points in high dimensions are pushed apart in the projection, preserving the separation between clusters.
The key t-SNE parameters to understand:
- perplexity: Controls the effective number of neighbors each point considers. Typical values range from 5 to 50. Low perplexity emphasizes very local structure and can fragment clusters; high perplexity incorporates more global information and merges clusters that are only locally distinct. For embedding visualization, perplexity values between 15 and 30 usually work well.
- n_iter: Number of optimization iterations. Usually 1000-2000 for convergence.
- learning_rate: Step size for gradient descent. The default "auto" setting in recent sklearn versions works well in most cases.
One critical warning: t-SNE distances are not interpretable. The distances between clusters in a t-SNE plot carry no meaningful information. Two clusters that look far apart in the t-SNE might be closer in the original space than two clusters that look adjacent. Only the local topology (which points cluster with which other points within a cluster) is meaningful. This warning is frequently ignored, leading to incorrect conclusions about global embedding structure.
UMAP: A Faster Alternative with Better Global Structure
Uniform Manifold Approximation and Projection (UMAP) is a newer alternative to t-SNE that has become increasingly popular for its combination of speed and structure preservation. It works by constructing a fuzzy topological representation of the data and optimizing a low-dimensional layout that approximates it. The theoretical foundation involves Riemannian geometry and algebraic topology, but the practical behavior is what matters for visualization.
UMAP has several practical advantages over t-SNE:
- Speed: UMAP scales much better to large datasets. For tens of thousands of word embeddings, t-SNE can take many minutes while UMAP finishes in seconds.
- Reproducibility: UMAP produces more consistent results across runs with the same random seed.
- Global structure: The relative positions of clusters in a UMAP projection are more geometrically meaningful than in t-SNE. If two word categories are semantically distant, they'll tend to be further apart in UMAP.
- Downstream use: UMAP can also be used as a preprocessing step for other algorithms because it better preserves global geometry.
The key UMAP parameters are:
- n_neighbors: Controls the balance between local and global structure, analogous to t-SNE's perplexity. Smaller values preserve local structure; larger values capture global relationships.
- min_dist: Controls how tightly points are packed in the projection. Smaller values allow UMAP to form tight clusters; larger values spread points out.
Let's visualize our structured embeddings with both methods:
from sklearn.manifold import TSNE
# Build the matrix of all embeddings
words = list(embeddings.keys())
vectors = np.array([embeddings[w] for w in words])
# Define semantic groups for coloring
groups = {
"Animals": ["cat", "kitten", "dog", "puppy", "horse"],
"Vehicles": ["car", "automobile", "truck", "bike", "bus"],
"Technology": ["computer", "laptop", "phone", "software", "program"],
}
word_to_group = {
word: group for group, wlist in groups.items() for word in wlist
}
group_colors = {
"Animals": "#E74C3C",
"Vehicles": "#3498DB",
"Technology": "#2ECC71",
}
# Apply t-SNE
tsne = TSNE(n_components=2, perplexity=4, random_state=42, max_iter=1000)
vectors_2d = tsne.fit_transform(vectors)
The visualization reveals semantic structure at a glance: animals, vehicles, and technology words form distinct clusters. Within each cluster, the most semantically similar words (cat/kitten, car/automobile) sit closest together. This is the kind of qualitative check that takes seconds to interpret and can catch problems that numerical scores would miss.
UMAP Visualization
try:
import umap
umap_available = True
except ImportError:
import subprocess
subprocess.run(
["uv", "pip", "install", "umap-learn"], check=True, capture_output=True
)
import umap
umap_available = True
reducer = umap.UMAP(
n_components=2, n_neighbors=4, min_dist=0.3, random_state=42
)
vectors_umap = reducer.fit_transform(vectors)
Both methods reveal the same underlying cluster structure. For production use with thousands of embeddings, UMAP's speed advantage becomes substantial. The choice between them often comes down to what structure you want to preserve: t-SNE for examining intra-cluster organization, UMAP for understanding inter-cluster relationships.
Neighbor Analysis as Qualitative Evaluation
Projection plots are useful for quick overview, but a complementary qualitative method is direct nearest-neighbor inspection. For any word of interest, retrieve the nearest neighbors by cosine similarity and read through them. This is often more diagnostic than a scatter plot because it lets you inspect specific cases rather than aggregate structure.
Good embeddings of "doctor" should have "physician", "nurse", "surgeon" as close neighbors. If you find "patient" very close, it might indicate the model is capturing co-occurrence patterns (doctors and patients frequently co-occur) rather than semantic similarity. If you find "Doctor" (capitalized) far away, your model might be case-sensitive in unexpected ways. These kinds of qualitative checks build intuition that purely numerical evaluations miss.