Recurrent Neural Networks (1995) › The Mathematics of Sequential Processing

While the conceptual idea behind RNNs is straightforward, understanding the mathematics reveals how they process sequences and learn from data. The core operation of an RNN can be expressed in a simple form: at each time step, the network takes the current input and the previous hidden state, combines them through a learned transformation, and produces both a new hidden state and an output.

Let us denote the hidden state at time step $t$ as $h_t$, the current input as $x_t$, and the output at time $t$ as $o_t$. The RNN updates its hidden state using the following equation:

$$h_t = f(U \cdot x_t + V \cdot h_{t-1} + b_h)$$

Here, $U$ is a weight matrix that transforms the current input $x_t$, and $V$ is a weight matrix that transforms the previous hidden state $h_{t-1}$. The term $b_h$ is a bias term that allows the network to shift its activations. The function $f$ is an activation function, typically the hyperbolic tangent tanh or the logistic sigmoid $\sigma$, which introduces nonlinearity into the transformation. This nonlinearity allows the network to learn complex patterns that linear functions cannot capture.

Once the hidden state is computed, the network produces an output using a second transformation:

$$o_t = g(W \cdot h_t + b_o)$$

In this equation, $W$ is a weight matrix that maps the hidden state to the output space, and $b_o$ is another bias term. The function $g$ is typically a softmax function when the output represents a probability distribution over discrete outcomes, such as predicting the next word in a vocabulary, or a linear function for continuous outputs.

One of the most important features of this formulation is that the weight matrices $U$, $V$, and $W$ are shared across all time steps. This means that the same parameters are used to process every element of the sequence, regardless of its position. This weight sharing serves two purposes. First, it dramatically reduces the number of parameters in the model, making it feasible to train on sequences of varying lengths without needing a different set of weights for each position. Second, it enables the network to generalize patterns learned at one time step to other time steps, which is essential for tasks like language modeling where the same grammatical structures can appear anywhere in a sentence.

The process of computing the hidden state at each time step is often called recurrent step or the forward pass through time. When we process an entire sequence, we repeatedly apply these equations, starting from an initial hidden state $h_0$ (which is often initialized to zeros) and stepping forward through the sequence one element at a time. This sequential processing is both the strength and the weakness of RNNs. It allows them to build up a representation of context over time, but it also means that processing long sequences can be slow, since each step depends on the previous one and cannot be computed in parallel.