Long Short-Term Memory (1997) › A Concrete Example: Tracking Subject-Verb Agreement

To make the LSTM's capabilities more concrete, consider how it might process a sentence with a grammatical dependency that spans multiple words. Take the sentence "The cat that chased the mice across the yard sat on the mat." Understanding this sentence requires maintaining information about grammatical relationships across significant distances.

When the LSTM encounters "The cat," the input gate learns to strongly incorporate this information into the cell state, recognizing that this represents the subject of the sentence. The cell state begins carrying the representation that we are tracking a singular subject performing some action.

As processing continues through "that chased the mice across the yard," the network faces a challenge. This relative clause provides important semantic information, but introduces its own grammatical structure that could be confusing. The plural "mice" and the various other nouns might mislead a simpler system into forgetting that the main subject is singular. However, the LSTM's forget gate has learned to preserve the information about the main subject in certain dimensions of the cell state while the input gate incorporates information about the relative clause in other dimensions. The cell state maintains both the fact that the main clause has a singular subject and the semantic details about what happened in the relative clause.

When the network finally reaches "sat," it needs to determine the correct verb form. Should it be "sat" (singular) or a hypothetical "sitted" if English worked differently, or would it be confused by the plural "mice" that appeared more recently? The LSTM's output gate at this point exposes the information about the main subject from the cell state, allowing the network to correctly predict the singular verb form despite the intervening relative clause with its plural object.

This example illustrates several key capabilities of LSTMs. They can selectively maintain information across long distances, distinguishing between what needs to be remembered long-term (the main subject) and what can be processed and partially forgotten (details of the relative clause). They can maintain multiple pieces of information simultaneously in different dimensions of the cell state. They can retrieve relevant information at the precise moment it's needed, even when many other words have intervened. These capabilities, impossible for traditional RNNs due to the vanishing gradient problem, made LSTMs effective for real language understanding tasks.