Recurrent Neural Networks (1995) › Processing Sequences in Practice

To make the abstract mathematics more concrete, consider how an RNN processes a simple sentence like "I love cats." The network processes this sentence one word at a time, updating its hidden state at each step and using that accumulated context to predict what comes next.

At the first time step, the network receives the word "I" as input. Since this is the beginning of the sequence, the hidden state starts as a zero vector or some other initialization. The network combines this initial hidden state with the input "I," applies the learned transformation, and produces a new hidden state that now carries information about the word "I." It also produces an output, which might be a probability distribution over the next word in the vocabulary. At this point, based only on seeing "I," the network might predict that the next word could be "am," "have," "will," or any number of verbs or auxiliary words that commonly follow the pronoun "I."

At the second time step, the network receives the word "love" as input. Now, instead of starting from scratch, it takes the hidden state from the previous step, which contains information about "I," and combines it with the new input "love." The network updates its hidden state to reflect the accumulated context of "I love," and produces a new prediction for the next word. This time, the predictions might include words like "you," "this," "cats," or other words that could plausibly follow "I love." The hidden state now captures the current word together with the history that preceded it.

At the third time step, the network processes "cats." It takes the hidden state from the previous step, which now encodes the context of "I love," and combines it with the input "cats." The hidden state is updated to represent "I love cats," and the network makes another prediction. At this point, it might predict punctuation like a period, or it might predict continuation words like "very" or "and," depending on what patterns it learned during training.

This step by step accumulation of context is what makes RNNs effective for sequential tasks. At each time step, the hidden state acts as a summary of everything the network has seen so far, allowing it to make informed predictions that take into account the entire history of the sequence up to that point. The network is not just looking at isolated words but is building up an understanding of how the words fit together, capturing grammatical structure, semantic relationships, and contextual dependencies. This ability to maintain and use context is what distinguishes RNNs from earlier approaches and makes them suitable for tasks like language modeling, where the meaning of a sentence depends on the order and combination of its words.