Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

RNN Architecture › Hidden State Dimensionality

Michael BrenndoerferMay 7, 20255 min read

Language AI HandbookAll chapters

Part 11 of 67Recurrent Neural Networks

Chapter 74RNN Architecture › Hidden State Dimensionality

Reading section 5 of 19 · Hidden State Dimensionality · 5 minSection 5 of 19 · 5 min

Section 5 of 19 · 5 min read

RNN Architecture › Hidden State DimensionalityOne section · Sections in this chapter5 / 19 ↓

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The size of the hidden state $H$ is a core hyperparameter. It controls the capacity of the RNN's memory. A larger $H$ gives the network more room to store information about the sequence, but it also increases the number of parameters and computational cost.

The dominant parameter cost comes from the recurrent matrix $\mathbf{W}_{hh} \in \mathbb{R}^{H \times H}$ , which scales quadratically with $H$ . Doubling the hidden size quadruples the number of recurrent parameters. For this reason, practical RNNs typically use hidden sizes between 128 and 2048, depending on the task and available compute.

The total parameter count for an Elman RNN is:

N_{\text{params}} = H^2 + H \cdot D + H + V \cdot H + V

where $H^2$ comes from $\mathbf{W}_{hh}$ , $H \cdot D$ from $\mathbf{W}_{xh}$ , $H$ from $\mathbf{b}_h$ , $V \cdot H$ from $\mathbf{W}_{hy}$ , and $V$ from $\mathbf{b}_y$ .

In[3]:

Code

# Compute RNN parameter counts for various hidden sizes
D = 128  # input (embedding) dimension
V = 10_000  # vocabulary size (for language modeling)

hidden_sizes = [64, 128, 256, 512, 1024]

param_counts = {}
for H in hidden_sizes:
    W_hh = H * H
    W_xh = H * D
    b_h = H
    W_hy = V * H
    b_y = V
    total = W_hh + W_xh + b_h + W_hy + b_y
    param_counts[H] = {
        "W_hh": W_hh,
        "W_xh": W_xh,
        "W_hy": W_hy,
        "total": total,
    }

Out[4]:

Console

     H        W_hh      W_xh        W_hy         Total
----------------------------------------------------
    64       4,096     8,192     640,000       662,352
   128      16,384    16,384   1,280,000     1,322,896
   256      65,536    32,768   2,560,000     2,668,560
   512     262,144    65,536   5,120,000     5,458,192
  1024   1,048,576   131,072  10,240,000    11,430,672

The quadratic growth from $\mathbf{W}_{hh}$ is visible in the numbers: going from $H = 64$ to $H = 512$ increases the recurrent matrix from 4,096 to 262,144 parameters, a 64-fold increase despite only an 8-fold increase in $H$ . This growth dynamic has practical consequences for how you budget parameters. For many language modeling tasks with large vocabularies, the output weight matrix $\mathbf{W}_{hy}$ is the single largest component at small hidden sizes, because $V \cdot H$ with $V = 10{,}000$ creates a 640,000-element matrix even at $H = 64$ .

Out[5]:

Visualization

Stacked bar chart showing RNN parameter distribution across hidden sizes 64 to 1024 with three colored component bars. — Stacked bar chart showing RNN parameter breakdown by component across five hidden sizes. The output weight matrix (blue) remains the dominant cost across this range because of the large vocabulary, while the recurrent matrix (orange) grows quadratically and takes an increasing share as the hidden size grows. Input weights (green) remain small throughout.

The stacked bars make the growth dynamic clear. Across the hidden sizes shown, the output weight matrix $\mathbf{W}_{hy}$ dominates because the vocabulary size ( $V = 10{,}000$ ) is large. The recurrent matrix $\mathbf{W}_{hh}$ does not overtake it in this range, but its share grows steadily because it scales quadratically with $H$ . This asymmetry explains why practitioners often use techniques like weight tying (sharing the embedding matrix with $\mathbf{W}_{hy}$ ) to reduce the output matrix cost.

Choosing Hidden Size in PracticeLink Copied

The choice of hidden size involves tradeoffs between representational capacity, parameter count, and computational speed. A rule of thumb is to start with hidden sizes of 128 or 256 for character-level models on small datasets, 512 to 1024 for word-level models on medium datasets, and 1024 to 2048 for larger-scale applications. However, the optimal size always depends on the complexity of the task, the amount of training data, and the available compute budget.

One important observation is that the hidden state does not need to be as large as the vocabulary to achieve good language modeling. The hidden state is a continuous vector, and even a 128-dimensional vector can encode rich probabilistic information about possible next tokens through the linear projection into vocabulary space. The compression that happens between $H$ -dimensional hidden state and $V$ -dimensional vocabulary logits is one of the key ways RNNs generalize: they must learn a compact representation of language that explains the data, rather than memorizing explicit transition probabilities.