Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

LSTM Gate Equations › LSTM Parameter Count

Michael BrenndoerferUpdated May 11, 20255 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 78LSTM Gate Equations › LSTM Parameter Count

Reading section 10 of 17 · LSTM Parameter Count · 5 minSection 10 of 17 · 5 min

Section 10 of 17 · 5 min read

LSTM Gate Equations › LSTM Parameter CountOne section · Sections in this chapter10 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding the parameter count is essential for estimating memory requirements and comparing architectures. Let's count every learnable parameter in an LSTM cell.

Each gate (forget, input, output) and the candidate cell state has:

One weight matrix $\mathbf{W} \in \mathbb{R}^{h \times d}$ for the input: $h \times d$ parameters
One weight matrix $\mathbf{U} \in \mathbb{R}^{h \times h}$ for the hidden state: $h \times h$ parameters
One bias vector $\mathbf{b} \in \mathbb{R}^h$ : $h$ parameters

Since we have four such components (forget gate, input gate, candidate, output gate), the total parameter count is:

\text{Parameters} = 4 \times (h \times d + h \times h + h) = 4h(d + h + 1)

where:

$h$ : the hidden dimension (size of the hidden and cell states)
$d$ : the input dimension (size of the input vector at each time step)
$h \times d$ : parameters in each input weight matrix $\mathbf{W}$
$h \times h$ : parameters in each recurrent weight matrix $\mathbf{U}$
$h$ : parameters in each bias vector $\mathbf{b}$
The factor of 4 accounts for the four components (forget gate, input gate, candidate, output gate)

For a more compact form, we can factor this as:

\text{Parameters} = 4h(d + h) + 4h

where $4h(d + h)$ counts all weight matrix parameters and $4h$ counts all bias parameters.

Example CalculationLink Copied

Let's compute the parameter count for a typical configuration with input dimension $d = 300$ (common for word embeddings) and hidden dimension $h = 512$ .

In[5]:

Code

d = 300  # input dimension (e.g., word embedding size)
h = 512  # hidden dimension

# Parameters per gate/candidate
params_W = h * d  # input weight matrix
params_U = h * h  # hidden state weight matrix
params_b = h  # bias vector
params_per_component = params_W + params_U + params_b

# Total parameters (4 components: forget, input, candidate, output)
total_params = 4 * params_per_component

Out[6]:

Console

Input dimension (d): 300
Hidden dimension (h): 512

Parameters per component:
  W matrix: 153,600 (512 × 300)
  U matrix: 262,144 (512 × 512)
  Bias: 512
  Subtotal: 416,256

Total LSTM parameters: 1,665,024

Using formula 4h(d + h + 1): 1,665,024

With these dimensions, a single LSTM layer has over 1.6 million parameters. For comparison, a vanilla RNN with the same dimensions would have:

\text{RNN Parameters} = h(d + h + 1) = 512 \times (300 + 512 + 1) = 416{,}256

where the single factor (instead of 4) reflects that a vanilla RNN has only one set of weights for its single transformation. The LSTM's four-fold increase in parameters comes from its four separate components (forget gate, input gate, candidate, output gate), each with their own weight matrices and biases.

Stacked LSTMsLink Copied

When you stack multiple LSTM layers, the parameter count increases. The first layer takes the input dimension $d$ , but subsequent layers take the hidden dimension $h$ as their input (since they receive the previous layer's hidden state).

In[7]:

Code

def lstm_params(d, h):
    """Calculate parameters for one LSTM layer."""
    return 4 * h * (d + h + 1)


# Stack of 3 LSTM layers
d = 300
h = 512
num_layers = 3

# First layer: input is d
first_layer_params = lstm_params(d, h)

# Subsequent layers: input is h
subsequent_layer_params = lstm_params(h, h)

total_stacked_params = (
    first_layer_params + (num_layers - 1) * subsequent_layer_params
)

Out[8]:

Console

3-layer stacked LSTM (d=300, h=512):
  Layer 1: 1,665,024 parameters
  Layer 2: 2,099,200 parameters
  Layer 3: 2,099,200 parameters
  Total: 5,863,424 parameters

A 3-layer LSTM with these dimensions has over 5.7 million parameters. Deep language models often use even larger hidden dimensions and more layers, quickly reaching hundreds of millions of parameters.

Out[9]:

Visualization

Line plot showing total LSTM parameters growing quadratically from 64 to 1024 hidden dimensions. — LSTM parameter count scaling with hidden dimension. The quadratic growth comes from the h×h recurrent weight matrices. Key parameter counts annotated for common hidden sizes.

Stacked area chart showing parameter breakdown into input weights, recurrent weights, and biases. — Parameter breakdown by type showing how recurrent weights (U matrices) dominate for larger hidden dimensions, while biases contribute negligibly.