LSTM Gate Equations › Key Parameters

When implementing or using LSTMs, several parameters significantly impact model behavior and performance:

• hidden_dim (hidden_size in PyTorch): The dimensionality of the hidden and cell states. Larger values increase model capacity but also increase computation and memory. Common values range from 128 to 1024, with 256-512 being typical for many NLP tasks.

• input_dim (input_size in PyTorch): The dimensionality of the input vectors at each time step. This is typically determined by your embedding layer or feature representation.

• num_layers: The number of stacked LSTM layers. Deeper networks can learn more complex patterns but are harder to train. Most applications use 1-3 layers; beyond 4-5 layers, gradient flow becomes problematic without techniques like residual connections.

• dropout: Regularization applied between LSTM layers (not within a single layer). Values of 0.2-0.5 are common. Note that dropout is only applied during training and when num_layers > 1.

• bidirectional: Whether to process sequences in both directions. Bidirectional LSTMs double the effective hidden size but cannot be used for autoregressive generation tasks.

• forget_bias (initialization): The initial value for the forget gate bias. Setting this to 1.0 or higher helps the network learn to preserve information early in training. This is often set implicitly by the framework but can be overridden.

• batch_first: In PyTorch, determines whether input tensors have shape (batch, seq, features) or (seq, batch, features). The former is more intuitive but the latter is the default for historical reasons.