Hierarchical Softmax › The Role of Node Vectors in Learned Representations

An aspect of hierarchical softmax that distinguishes it from standard softmax is the nature of its learnable parameters. Standard softmax has two parameter matrices: an input embedding matrix $W_{in}$ for center words and an output embedding matrix $W_{out}$ for context words. After training, you typically use $W_{in}$ as the word representations, or average $W_{in}$ and $W_{out}$.

Hierarchical softmax replaces $W_{out}$ with a matrix of node vectors $W_{node}$, one per internal node. These node vectors are not word embeddings in the usual sense: they do not correspond to specific vocabulary words and are not typically used as word representations after training. Their role is purely to parameterize the binary decisions in the tree. Each node vector learns to represent the split between the two halves of the vocabulary it divides, which makes it a kind of "discriminative boundary" rather than a "semantic embedding."

This distinction matters for practical use. After training with hierarchical softmax, the word embeddings you extract are the input embeddings from $W_{in}$. The node vectors serve their purpose during training and are then discarded. Unlike negative sampling, which produces two embedding matrices (both of which can potentially be used as word representations), hierarchical softmax produces only one directly useful embedding matrix.

There is a subtle benefit to this design. Because $W_{in}$ is the only embedding matrix updated via the center word gradient, center word embeddings receive a single clear gradient signal from each training pair. In standard softmax and negative sampling, both input and output embeddings are updated for every word involved in a training pair, which can introduce noise. The well-defined separation in hierarchical softmax between "word embeddings" and "node vectors" can lead to more stable training dynamics in some settings.