Memory Networks (2014) › The Solution: Give Neural Networks a Library Card

Memory Networks placed a structured external memory beside the neural network instead of compressing every fact into its parameters. The network then learned how to find relevant facts in that memory when needed.

Think of it like the difference between memorizing an encyclopedia versus knowing how to use a library. Memorizing the encyclopedia (the old neural network approach) seems powerful until you realize you can only memorize so much, and updating your knowledge means re-memorizing everything. Using a library (the Memory Networks approach) means you store information externally and develop the skill of finding what you need when you need it. The library can grow indefinitely, you can add new books without forgetting old ones, and you can locate specific information efficiently through a good indexing system.

The Memory Networks architecture embodied this library metaphor through four components. The input module acted like a reference librarian, converting incoming questions into a form suitable for search. The memory module was the library itself: structured storage for facts or documents. The output module gathered relevant material, and the response module formatted the answer.

The Input Module: Converting Questions to Queries

When you ask a librarian for help, the librarian interprets the question and formulates search terms that will find relevant material. The input module performed the corresponding operation in a Memory Network.

The module converted natural-language questions into dense vector representations: lists of numbers encoding aspects of the question's meaning. For "What is the capital of France?", the vector represented the query about a capital city and France. The system used this vector to search memory, much like search terms in a library catalog.

The encoding process used neural networks trained to capture semantic meaning. Questions asking about the same thing in different words ("What is France's capital?" vs "What city is the capital of France?") would produce similar vector representations, enabling the system to find relevant information regardless of how the question was phrased.

The Memory Module: The Library Itself

The memory module held information without compressing it into neural-network weights. Like a library collection, it organized items so the system could locate specific information.

Each piece of information occupied a memory slot, a discrete location that could hold a fact or a passage. One slot might store "Paris is the capital of France" and another "The Eiffel Tower is located in Paris." These entries were encoded as dense vectors rather than retained as raw text. Together they formed a memory matrix in which each row represented one stored item.

Vector encoding made the memory searchable. The system compared the query vector from the input module with each memory slot to find the closest matches. Slots semantically related to the question occupied nearby regions of the representation space.

The memory could be initialized with any knowledge source you wanted the system to access. Load a document collection, and each document (or paragraph, or sentence) becomes a memory slot. Initialize with a knowledge base of facts, and each fact gets its own slot. Unlike standard neural networks that needed to compress everything during training, Memory Networks could work with external knowledge directly, accessing it during inference without modification to the network weights.

The Output Module: Attention as a Spotlight

Memory Networks used attention to access their external memory. Rather than retrieve one slot or process the entire memory equally, the output module formed a weighted combination of memory contents. In the spotlight analogy, the brightness assigned to each source indicated its relevance.

The attention mechanism worked by computing a relevance score for each memory slot. Given the query vector from the input module, the system measured how semantically similar each memory slot was to the query. Memory slots highly relevant to the question received high attention weights. Irrelevant slots received weights close to zero. These weights determined how much each memory slot contributed to the final answer.

For the question "What is the capital of France?" the system might assign high attention weight to the memory slot containing "Paris is the capital of France," moderate weight to "The Eiffel Tower is located in Paris" (relevant but not directly answering the question), and near-zero weight to "Tokyo is the capital of Japan" (irrelevant despite being about capitals).

Because attention was differentiable, gradients could pass through it during end-to-end training with backpropagation. The model learned which memory slots were relevant to each question from the training objective.

The Response Module: Formatting the Answer

The final component took the information retrieved from memory and formatted it into an appropriate answer. Depending on the task, this might mean producing a single word, selecting from multiple choice options, or generating a complete sentence. The response module learned to extract the key information from the retrieved memory contents and present it in the expected format.

If the question was "What is the capital of France?" and the retrieved information included "Paris is the capital of France," the response module would extract "Paris" as the answer. For a more complex question requiring synthesis of multiple facts, the response module would combine information from several high-attention memory slots into a coherent response.

Memory Networks supported multi-hop reasoning, in which an answer required several retrieval steps. The earlier question, "What is the capital of the country where the Eiffel Tower is located?", cannot be answered with one lookup.

Memory Networks handled this through iterative attention updates. In the first reasoning hop, the system would use the question to query memory and identify relevant information. For our example, the first hop might retrieve "The Eiffel Tower is located in Paris, France" by attending to memory slots discussing the Eiffel Tower. This gives us France as an intermediate result.

In the second hop, the system would use both the original question and the information retrieved in the first hop to refine its attention. The resulting query asks "What is the capital of France?", leading it to the memory slot containing "Paris is the capital of France."

This process could continue across additional hops. Each hop updated the attention distribution using information retrieved so far, allowing the model to move from the initial question through intermediate facts toward an answer.

The multi-hop mechanism addressed one of the fundamental limitations of standard neural networks. Rather than trying to maintain all intermediate information in fixed-size hidden states, Memory Networks could explicitly retrieve intermediate results from memory, use them to guide the next retrieval, and build up the answer step by step. The external memory acted as a scratch pad for reasoning, storing intermediate results that could be accessed in later steps.

Training optimized the question encoder, memory attention, and answer generator together. The objective therefore covered the full retrieval-and-reasoning process rather than only the final output layer.

The training data consisted of question-answer pairs and the memory contents needed to answer them. For example, a geography memory could accompany the question "What is the capital of France?" and its answer, "Paris." The system did not receive explicit supervision about which memory slots to attend to; it inferred that relationship from the examples.

The entire system was differentiable, so gradients could flow from the final answer through the attention mechanism to the question encoder. When the model produced an incorrect answer, backpropagation adjusted the response module and the representations used to query memory. Training thereby associated useful attention patterns with correct answers.

Joint training allowed the components to adapt together. The question encoder produced query vectors that exposed relevant slots, while attention assigned higher weights to useful contents. The response module learned to extract and format the retrieved information.