Agent Memory Systems: From Context to Persistent Storage › Memory Summarization: Compression and Consolidation

Raw conversation logs and tool outputs consume tokens rapidly. A detailed hour-long interaction might exhaust the context window of even the most capable models. Memory summarization compresses this history into dense representations that preserve needed information while discarding noise. This process is analogous to sleep-dependent memory consolidation in humans, where the brain transfers information from the hippocampus to the cortex, extracting patterns and discarding ephemeral details.

Without summarization, agents face a painful trade-off: either truncate the conversation history, losing early context entirely, or include everything and hit token limits that prevent new reasoning. Summarization offers a middle path, distilling the essence of long interactions into compact forms that fit within constraints while retaining actionable information. A well-summarized interaction might compress ten thousand tokens of dialogue into two hundred tokens that capture key decisions and preferences alongside relevant facts, freeing the remaining budget for current reasoning.

The simplest approach maintains a rolling summary. After every $N$ turns or $T$ tokens, the agent processes the recent history through a summarization prompt:

This summary replaces the raw text in the context window, freeing space for new interactions. The agent might keep the most recent five turns in full detail while summarizing everything before that point into a condensed paragraph. This creates a sliding window of high-fidelity recent memory backed by increasingly abstracted historical context.

However, aggressive summarization risks losing necessary details, creating a trade-off between compression ratio and information fidelity. A summary stating "we discussed authentication" loses the nuance that "we specifically decided to use JWT with a custom refresh strategy rather than session cookies." Some systems address this by maintaining multiple summary levels: a detailed summary of the last ten turns, a medium-level summary of the last fifty turns, and a high-level abstract of everything before that. When retrieving context, the agent can zoom in or out depending on the specificity required. This multi-resolution approach mirrors how human memory stores both the gist of an experience (we went on vacation to Italy) and specific vivid details (the taste of gelato in a particular piazza).

The quality of summarization depends heavily on the instructions given to the summarizer. Generic prompts produce generic summaries that might omit domain-specific details the agent needs. A coding agent benefits from prompts that explicitly ask to preserve function signatures, error messages, and architectural decisions. A customer service agent needs prompts that capture complaint categories, resolution status, and customer sentiment. Tailoring the summarization prompt to the domain can sharply improve how much actionable information survives compression.

More advanced systems organize memory hierarchically, similar to how computer memory uses caches and RAM. MemGPT, for instance, implements a tiered architecture that explicitly manages information flow between storage layers with different capacity and latency characteristics.

The tiers form a spectrum from fastest-and-smallest to slowest-and-largest:

Core memory holds always-present system instructions and user preferences, analogous to CPU registers. This small, high-speed store contains the most necessary information that the agent needs for every turn: the user's name, the current project context, and any hard constraints that must never be violated. Core memory is never evicted and is always included in the context window without requiring any retrieval operation.

Working context contains the recent conversation within the context window, analogous to RAM. This holds the active conversation thread and recent tool outputs in their original form. It gives high-fidelity access to recent history but is limited by the context window size. When working context fills up, older entries must be either discarded or moved to longer-term storage.

External storage archives conversations and documents on disk-equivalent storage. This large, slow repository holds the full history of past interactions, reference documents, and background knowledge. Retrieval from this tier requires explicit search operations using the vector similarity and multi-factor scoring techniques described earlier.

The agent explicitly manages movement between tiers through function calls: search_archives(query), add_to_core(key, value), and summarize_and_store(segment). This explicit memory management, while adding complexity, allows the agent to reason about its own memory constraints and optimize information placement strategically. For example, when learning a necessary API key or user constraint, the agent can explicitly move it into core memory for immediate access, while archiving routine conversation to external storage. This meta-cognitive capacity, the ability to think about and manage one's own memory, is one of the more intriguing capabilities of advanced agent architectures.

Rather than compressing text into shorter text, some systems extract structured knowledge graphs from conversation history. The agent parses conversations into subject-relation-object triples. "Alice prefers Python over Java" becomes the triple (Alice, prefers, Python) and the triple (Alice, avoids, Java). This transformation converts unstructured narrative into structured data that supports precise querying and logical inference.

These triples populate a graph database where nodes stand for entities and edges stand for relationships. Retrieval then involves graph traversal: finding Alice's node and following preference edges yields her language choices without scanning conversation logs. If you ask, "What programming languages does Alice like?", the system queries the graph for all objects connected to "Alice" via "prefers" edges, returning a clean list without wading through paragraphs of dialogue.

This approach excels for factual, relational knowledge but struggles with procedural information and fine-grained context that resists triple extraction. Statements like "Alice prefers Python for data science but uses JavaScript for web development" decompose cleanly into conditional triples, but procedural knowledge like "When debugging async code, Alice starts by checking for race conditions" loses its conditional structure when flattened into simple relations. The temporal dimension is also lost: a knowledge graph stores that Alice prefers Python, but cannot naturally stand for that she used to prefer Ruby and switched six months ago after a frustrating project.

Hybrid systems often combine graph extraction for entity relationships with narrative summarization for procedural and contextual information. The graph handles the "who knows what" questions efficiently, while summarized narrative handles the "how and why things happened" questions that require more context to answer correctly. This combination uses the precision of structured data where that precision is achievable, and falls back to the flexibility of natural language where it is not.