Hierarchical Softmax › Key Parameters

The key configuration choices for hierarchical softmax are:

• Tree construction method: Huffman coding is the standard choice, as it minimizes expected path length by assigning shorter paths to frequent words. A balanced binary tree is simpler to implement but ignores frequency information entirely and performs significantly worse on skewed vocabularies.
• Embedding dimension: The dimensionality of both input embeddings and node vectors. Typical values are 100 to 300 for word2vec-style models. Larger dimensions can capture more semantic nuance but require more memory and training time. The node vectors must match the embedding dimension.
• Learning rate: Controls the step size for gradient updates along paths. Typical starting values range from 0.025 to 0.05, often decayed linearly during training. Because hierarchical softmax updates only a small subset of parameters per step, it can tolerate slightly higher learning rates than methods that update all parameters globally.
• Window size: The context window determines which (center, context) pairs are generated for training. Larger windows (5 to 10 words) capture broader semantic context and tend to produce embeddings useful for topic-level similarity. Smaller windows (1 to 3 words) capture tighter syntactic relationships and often produce embeddings better suited for analogy tasks.
• Subsampling of frequent words: In practice, very frequent words like "the" and "a" are subsampled during training (randomly discarded from context windows with a probability proportional to their excess frequency). This prevents them from dominating gradient updates and improves embedding quality for less frequent words. Subsampling is equally applicable with hierarchical softmax and negative sampling.