Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

FastText › Visualizing the Learned Embedding Space

Michael BrenndoerferUpdated March 10, 20264 min read

Language AI HandbookAll chapters

Part 4 of 67Word Embeddings

Chapter 26FastText › Visualizing the Learned Embedding Space

Reading section 8 of 15 · Visualizing the Learned Embedding Space · 4 minSection 8 of 15 · 4 min

Section 8 of 15 · 4 min read

FastText › Visualizing the Learned Embedding SpaceOne section · Sections in this chapter8 / 15 ↓

Sections in this chapter15

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A PCA projection of the trained embeddings shows whether morphological families ended up near each other in the full embedding space. PCA finds the directions of maximum variance in the embedding matrix and projects each vector onto the first two principal components, producing a 2D view that preserves as much structure as possible from the original 30-dimensional space.

In[22]:

Code

from sklearn.decomposition import PCA

viz_words = [
    "run",
    "running",
    "runner",
    "runing",
    "walk",
    "walking",
    "walker",
    "walked",
    "swim",
    "swimming",
    "swimmer",
    "swiming",
    "jump",
    "jumping",
    "jumper",
    "jumped",
]

# The fourth form in each family is absent from the training vocabulary.
oov_in_viz = {w for w in viz_words if w not in in_vocab_set}

emb_list, valid_words = [], []
for w in viz_words:
    e = model.get_embedding(w)
    if norm(e) > 0:
        emb_list.append(e)
        valid_words.append(w)

emb_arr = np.array(emb_list)
coords_2d = PCA(n_components=2).fit_transform(emb_arr)

family_colors = {
    "run": theme_color("#1f77b4"),
    "running": theme_color("#1f77b4"),
    "runner": theme_color("#1f77b4"),
    "runing": theme_color("#1f77b4"),
    "walk": theme_color("#2ca02c"),
    "walking": theme_color("#2ca02c"),
    "walker": theme_color("#2ca02c"),
    "walked": theme_color("#2ca02c"),
    "swim": theme_color("#d62728"),
    "swimming": theme_color("#d62728"),
    "swimmer": theme_color("#d62728"),
    "swiming": theme_color("#d62728"),
    "jump": theme_color("#9467bd"),
    "jumping": theme_color("#9467bd"),
    "jumper": theme_color("#9467bd"),
    "jumped": theme_color("#9467bd"),
}

Out[23]:

Visualization

2D scatter plot showing morphological word families clustered together after FastText training with OOV words marked with asterisks. — PCA projection of FastText embeddings to two dimensions. Each color identifies one morphological family, while square points and asterisks identify the unseen forms runing, walked, swiming, and jumped. The OOV forms remain near relatives from the same family because their n-grams encode the shared root.

OOV words land within or near the correct family regions despite never having been seen during training. The PCA projection confirms that the n-gram sharing mechanism is doing meaningful work: structural similarity in character sequences translates into proximity in the learned embedding space. When the model has never seen "runing" as a complete token, it can still place it near "run," "running," and "runner" because most of its n-grams are familiar from training on related words.

PCA preserves some structure and loses some. The first two principal components capture the directions of maximum variance, which in this case align with the between-family separation axis and within-family variation. The actual embedding space is 30-dimensional, and some structure that exists there is invisible in the 2D projection. The clusters you see are real, but the distances between clusters in 2D are not perfectly representative of distances in 30 dimensions.