Recurrent Neural Networks (1995) › The Architecture of Memory

To understand how RNNs work, it helps to start with what makes them different from the feedforward networks that came before. In a standard feedforward network, each layer of neurons passes its output to the next layer, and information flows straight through from input to output. Once the network has processed an input and produced an output, it has no record of what happened. The next input is processed from scratch, as if the network had never seen anything before.

RNNs change this by introducing recurrent connections, which are edges that loop back into the same layer or to previous layers in the network. These connections create cycles in the computational graph, allowing the network to maintain state across time steps. When an RNN processes a new input, it combines that input with the hidden state from the previous time step, effectively giving the network access to its own recent history. This hidden state acts as the network's memory, a numerical representation that captures relevant information from everything the network has seen so far in the sequence.

The hidden state is updated at every time step as the RNN processes each new element of the sequence. It is transformed by a learned function that takes both the current input and the previous hidden state as arguments, producing a new hidden state that is passed forward to the next time step. In this way, the hidden state serves as a bridge between past and present, allowing information to flow through time. The network can use this accumulated context to make more informed predictions about what comes next, whether that is the next word in a sentence, the next phoneme in a spoken utterance, or the next value in a time series.

This architecture solves the context problem that plagued earlier networks. Instead of treating each input as an isolated event, the RNN builds up a representation of the sequence as it goes, updating its internal memory with each new piece of information. The result is a network that can perform tasks that require understanding sequences, such as predicting the next word in a sentence, recognizing spoken words from sequences of audio features, or identifying patterns in time series data. The hidden state makes all of this possible, providing a compact summary of the sequence history that the network can use to inform its predictions.