Memory Networks (2014) › 2014: Memory Networks: Teaching Neural Networks to Remember

In 2014, Jason Weston and his team at Facebook AI Research published a paper that asked a simple question: what if neural networks could have a library? Instead of relying only on knowledge compressed into their weights, they could store facts externally and retrieve relevant information when answering questions.

The idea may sound familiar today because retrieval-augmented generation (RAG) systems routinely combine language models with external databases. In 2014, however, neural networks could only retain what fit in their parameters. A network trained on a document collection had to compress that collection into its weights. Adding new information required retraining, and increasing the number of facts strained the model's fixed capacity.

Memory Networks addressed this constraint with explicit external memory that a neural network could read from and write to during reasoning. The architecture separated memory storage from the reasoning components. Attention mechanisms selected relevant information from a knowledge base that did not have to fit in the network parameters, including information needed across multiple reasoning steps.

The design applied beyond question answering: knowledge could remain external while the model learned how to retrieve it. This separation later appeared in reading-comprehension systems and RAG architectures for knowledge-intensive applications. Memory Networks provided an early example of combining neural learning with external knowledge access instead of increasing parameter capacity alone.