Hierarchical Softmax › The Training Objective

The hierarchical softmax objective follows directly from maximum likelihood. We want to maximize the probability of the observed (center, context) pairs in the training corpus. Taking the log converts products into sums, which is numerically more stable and mathematically easier to differentiate:

$$\mathcal{L} = \sum_{(w_c, w_o) \in \text{corpus}} \log P(w_o | \mathbf{h}_{w_c})$$

where:

• $w_c$: the center word
• $w_o$: the observed context word
• $\mathbf{h}_{w_c}$: the input embedding of center word $w_c$
• $P(w_o | \mathbf{h}_{w_c})$: computed as the product of path probabilities

Substituting the path probability formula and applying the logarithm:

$$\log P(w_o | \mathbf{h}) = \sum_{j=1}^{L(w_o)-1} \log \sigma\!\left( d_j \cdot \mathbf{v}_{n(w_o, j)} \cdot \mathbf{h} \right)$$

where:

• $\log P(w_o | \mathbf{h})$: log-probability of context word $w_o$
• $d_j$: direction code at step $j$ along the path to $w_o$
• $\mathbf{v}_{n(w_o, j)}$: the learned node vector at step $j$

The product of probabilities becomes a sum of log-probabilities, which eliminates the risk of underflow for long paths. Converting the maximization to minimization (for gradient descent), we minimize the negative log-likelihood:

$$\mathcal{J} = -\sum_{j=1}^{L(w_o)-1} \log \sigma\!\left( d_j \cdot \mathbf{v}_{n(w_o, j)} \cdot \mathbf{h} \right)$$

This loss has a pleasant interpretation that makes the training dynamics intuitive. Each term $-\log \sigma(d_j \cdot \mathbf{v}_j \cdot \mathbf{h})$ is the binary cross-entropy loss for a single binary classifier at node $j$. The "correct label" at each node is determined by the direction $d_j$: if we should go left, the correct output is $\sigma > 0.5$, which corresponds to a positive dot product. If we should go right, the correct output is $\sigma < 0.5$, corresponding to a negative dot product. The loss penalizes the model whenever any of these binary classifiers makes a wrong or uncertain decision.

Training pushes the model to make each binary decision more confidently correct. Concretely, for each node on the path to the target word, the node vector $\mathbf{v}_j$ and the context embedding $\mathbf{h}$ are adjusted so that their dot product has the correct sign and magnitude. After sufficient training, the model should be able to traverse the tree decisively, assigning high probability to each correct turn and arriving at the target leaf with high confidence.