RNN Architecture › The Elman RNN Formulation

The most basic RNN, introduced by Jeffrey Elman in 1990, computes the hidden state and output at each time step with two equations.

The hidden state update is:

$$\mathbf{h}_t = \tanh\!\left(\mathbf{W}_{hh}\,\mathbf{h}_{t-1} + \mathbf{W}_{xh}\,\mathbf{x}_t + \mathbf{b}_h\right)$$

where:

• $\mathbf{h}_t \in \mathbb{R}^{H}$: the hidden state at time step $t$, a vector of dimension $H$ (the hidden size)
• $\mathbf{h}_{t-1} \in \mathbb{R}^{H}$: the hidden state from the previous time step (initialized to a zero vector at $t=0$)
• $\mathbf{x}_t \in \mathbb{R}^{D}$: the input at time step $t$, a vector of dimension $D$ (the input size)
• $\mathbf{W}_{hh} \in \mathbb{R}^{H \times H}$: the recurrent weight matrix connecting the previous hidden state to the current one
• $\mathbf{W}_{xh} \in \mathbb{R}^{H \times D}$: the input weight matrix connecting the current input to the hidden state
• $\mathbf{b}_h \in \mathbb{R}^{H}$: the hidden state bias vector
• $\tanh$: the hyperbolic tangent activation function, applied element-wise to squash the values to $(-1, 1)$

The output at time step $t$ (when producing a prediction at that step) is:

$$\mathbf{y}_t = \mathbf{W}_{hy}\,\mathbf{h}_t + \mathbf{b}_y$$

where:

• $\mathbf{y}_t \in \mathbb{R}^{V}$: the output vector at time step $t$ (often called logits before softmax)
• $\mathbf{W}_{hy} \in \mathbb{R}^{V \times H}$: the output weight matrix
• $\mathbf{b}_y \in \mathbb{R}^{V}$: the output bias vector
• $V$: the output size (e.g., vocabulary size for language modeling)

The two equations capture the complete computation of an RNN. The first equation is the memory update: it blends the current input with the running history. The second equation is the prediction: it projects the memory into output space. Separating memory from prediction makes the architecture conceptually tidy and practically modular.

The $\tanh$ activation is the traditional choice for the hidden state. It maps large positive values toward $+1$ and large negative values toward $-1$, which helps keep the hidden state within a bounded range. As we will see in the chapter on Vanishing Gradients, $\tanh$ also has desirable gradient properties compared to unbounded activations.

You might wonder why RNNs use $\tanh$ rather than ReLU, which works well in feedforward networks. The answer lies in the recurrence. In a feedforward network, each layer is applied once. In an RNN, the same $\mathbf{W}_{hh}$ matrix is multiplied at every time step. If the hidden state can grow without bound (as it would with ReLU and spectral radius greater than 1), gradients explode. The $\tanh$ squashing function forces the hidden state to remain in $(-1, 1)^H$, which dampens but does not eliminate the explosion problem.

There is a deeper issue with ReLU in recurrent settings. The derivative of ReLU is either 0 or 1, which means that when flowing gradients backward through many steps, you either pass them unchanged (if all units are active) or kill them entirely (if units die). The $\tanh$ derivative is always positive, between 0 and 1, meaning gradients shrink but never vanish abruptly. This gives the training process a somewhat more predictable gradient signal, although the shrinking still accumulates to the vanishing gradient problem over long sequences.

Some research has explored using identity initialization (initializing $\mathbf{W}_{hh}$ to the identity matrix) combined with ReLU in what are called IndRNNs (Independently Recurrent Neural Networks). By constraining each hidden unit to only connect to itself in the recurrent step, the instability from matrix multiplication is avoided. But the vanilla Elman RNN with $\tanh$ remains the standard baseline, and the Vanishing Gradients chapter explores this tension in detail and explains why LSTMs and GRUs were designed to address it.

At the very first time step, there is no previous hidden state. The convention is to initialize $\mathbf{h}_0 = \mathbf{0}$: a zero vector of dimension $H$. This is the most common choice, though it is not the only one.

In some settings, the initial hidden state is learned as a parameter. This is particularly useful when the model is conditioned on a context that should influence the entire sequence, such as a class label or a conditioning image embedding. In encoder-decoder architectures (which Part XII covers), the encoder's final hidden state becomes the decoder's initial hidden state, and this transfer of context is the mechanism by which the encoder communicates its understanding of the input to the decoder.

Zero initialization is equivalent to saying "the network starts with no information about the sequence." Whether this is the right inductive bias depends on the task. For language modeling over independent documents, it is perfectly fine: each document starts fresh. For modeling a stream of data with continuity across chunks, you may want to carry the final hidden state from one batch forward as the initial state of the next batch. This technique, called stateful RNN processing or truncated BPTT, is covered in more detail in the Backpropagation Through Time chapter.

In practice, many deep learning libraries (including PyTorch) compute the hidden state update using a single matrix multiplication rather than two:

$$\mathbf{h}_t = \tanh\!\left(\mathbf{W}\,[\mathbf{h}_{t-1};\,\mathbf{x}_t] + \mathbf{b}_h\right)$$

where $\mathbf{W} \in \mathbb{R}^{H \times (H + D)}$ is the horizontal concatenation of $\mathbf{W}_{hh}$ and $\mathbf{W}_{xh}$, and $[\mathbf{h}_{t-1};\,\mathbf{x}_t]$ is the vertical concatenation of the two vectors. This single matrix-vector product is computationally equivalent to the two-matrix formulation but more efficient to execute as a single GPU kernel call.