Implementing Memory in Our Agent › Starting Point: A Complete Memory System

Michael BrenndoerferJuly 5, 20255 min read
AI Agent HandbookAll chapters
Part 6 of 15Memory and Retrieval - How Agents Remember
Chapter 17Implementing Memory in Our Agent › Starting Point: A Complete Memory System
Reading section 2 of 13 · Starting Point: A Complete Memory System · 5 minSection 2 of 13 · 5 min

Book contents

Section 2 of 13 · 5 min read

Implementing Memory in Our Agent › Starting Point: A Complete Memory SystemOne section · Sections in this chapter2 / 13 ↓

Let's build our assistant step by step. We'll start with the core components and then assemble them into a working system.

The Conversation Manager

First, we need something to handle short-term memory. This manages the ongoing conversation:

In[3]:
Code
class ConversationManager:
    """
    Manages short-term conversation memory with automatic windowing.
    """
    
    def __init__(self, max_messages: int = 20):
        self.messages = []
        self.max_messages = max_messages
    
    def add_user_message(self, content: str):
        """Add a user message to history."""
        self.messages.append({
            "role": "user",
            "content": content
        })
        self._trim_if_needed()
    
    def add_assistant_message(self, content: str):
        """Add an assistant message to history."""
        self.messages.append({
            "role": "assistant",
            "content": content
        })
        self._trim_if_needed()
    
    def _trim_if_needed(self):
        """Keep only the most recent messages."""
        if len(self.messages) > self.max_messages:
            self.messages = self.messages[-self.max_messages:]
    
    def get_messages(self) -> list:
        """Get all current messages."""
        return self.messages.copy()
    
    def clear(self):
        """Start a fresh conversation."""
        self.messages = []

This is straightforward. We maintain a list of messages and automatically trim it when it gets too long. The sliding window approach keeps costs predictable and prevents context overflow.

The Knowledge Store

Next, we need long-term memory. This stores facts persistently:

In[4]:
Code
import json
import os
from datetime import datetime

class KnowledgeStore:
    """
    Stores and retrieves long-term facts about the user.
    """
    
    def __init__(self, storage_file: str = "knowledge.json"):
        self.storage_file = storage_file
        self.facts = self._load_facts()
    
    def _load_facts(self) -> list:
        """Load facts from disk."""
        if os.path.exists(self.storage_file):
            with open(self.storage_file, 'r') as f:
                return json.load(f)
        return []
    
    def _save_facts(self):
        """Save facts to disk."""
        with open(self.storage_file, 'w') as f:
            json.dump(self.facts, f, indent=2)
    
    def add_fact(self, fact: str, category: str = None):
        """Store a new fact."""
        entry = {
            "fact": fact,
            "category": category,
            "timestamp": datetime.now().isoformat()
        }
        self.facts.append(entry)
        self._save_facts()
    
    def search(self, query: str) -> list:
        """
        Search for relevant facts.
        Simple keyword matching for now.
        """
        query_lower = query.lower()
        results = []
        
        for entry in self.facts:
            fact_lower = entry["fact"].lower()
            if query_lower in fact_lower:
                results.append(entry["fact"])
            elif entry["category"] and query_lower in entry["category"].lower():
                results.append(entry["fact"])
        
        return results
    
    def get_all_facts(self) -> list:
        """Get all stored facts."""
        return [entry["fact"] for entry in self.facts]

This provides persistent storage with simple keyword search. We save facts to a JSON file so they survive between sessions. Each fact has a timestamp and optional category for organization.

Putting It Together

Now let's combine these into a complete assistant. Example (Claude Sonnet 4.5):

In[5]:
Code
import os
from anthropic import Anthropic

class PersonalAssistant:
    """
    A personal assistant with both short-term and long-term memory.
    """
    
    def __init__(self, api_key: str):
        # Using Claude Sonnet 4.5 for its excellent agent reasoning capabilities
        self.client = Anthropic(api_key=api_key)
        self.model = "claude-sonnet-4-5"
        
        # Memory systems
        self.conversation = ConversationManager(max_messages=20)
        self.knowledge = KnowledgeStore()
    
    def chat(self, user_message: str) -> str:
        """
        Process a user message and return a response.
        Handles both conversation and knowledge storage.
        """
        # Check if this is a memory command
        if self._is_memory_command(user_message):
            return self._handle_memory_command(user_message)
        
        # Search for relevant facts
        relevant_facts = self.knowledge.search(user_message)
        
        # Build system prompt with context
        system_prompt = self._build_system_prompt(relevant_facts)
        
        # Add user message to conversation
        self.conversation.add_user_message(user_message)
        
        # Get response from Claude
        response = self.client.messages.create(
            model=self.model,
            max_tokens=1024,
            system=system_prompt,
            messages=self.conversation.get_messages()
        )
        
        # Extract and store response
        assistant_message = response.content[0].text
        self.conversation.add_assistant_message(assistant_message)
        
        return assistant_message
    
    def _is_memory_command(self, message: str) -> bool:
        """Detect if user wants to store something."""
        keywords = ["remember", "store", "save", "keep in mind", "note that"]
        message_lower = message.lower()
        return any(keyword in message_lower for keyword in keywords)
    
    def _handle_memory_command(self, message: str) -> str:
        """Extract and store information."""
        # Use Claude to extract the fact
        response = self.client.messages.create(
            model=self.model,
            max_tokens=256,
            system="Extract the key fact the user wants you to remember. Return only the fact as a clear, concise statement.",
            messages=[{"role": "user", "content": message}]
        )
        
        fact = response.content[0].text
        self.knowledge.add_fact(fact)
        
        return f"Got it! I'll remember that {fact.lower()}"
    
    def _build_system_prompt(self, relevant_facts: list) -> str:
        """Build system prompt with retrieved knowledge."""
        base_prompt = "You are a helpful personal assistant."
        
        if not relevant_facts:
            return base_prompt
        
        facts_text = "\n".join(f"- {fact}" for fact in relevant_facts)
        return f"""{base_prompt}

You have access to the following information about the user:

{facts_text}

Use this information when relevant to provide personalized responses."""
    
    def start_new_conversation(self):
        """Clear conversation history but keep long-term knowledge."""
        self.conversation.clear()

Let's see this in action:

In[6]:
Code
## Create the assistant
assistant = PersonalAssistant(api_key=os.getenv("ANTHROPIC_API_KEY"))

## Store some facts
print(assistant.chat("Remember that my birthday is July 20"))
## Output: Got it! I'll remember that your birthday is july 20

print(assistant.chat("Remember that I'm allergic to peanuts"))
## Output: Got it! I'll remember that you're allergic to peanuts

print(assistant.chat("Remember that I prefer Italian food"))
## Output: Got it! I'll remember that you prefer italian food

## Now ask questions that use this knowledge
print(assistant.chat("What should I be careful about when eating out?"))
## Output: Based on what I know, you should be careful about peanuts since 
## you're allergic to them. Always inform restaurant staff about your allergy 
## and ask about ingredients, especially in sauces and desserts where peanuts 
## might be hidden...

print(assistant.chat("Suggest a restaurant for my birthday dinner"))
## Output: For your birthday on July 20, I'd suggest an Italian restaurant 
## since you prefer Italian food. Make sure to mention your peanut allergy 
## when making the reservation...
Out[6]:
Console
Got it! I'll remember that your birthday is july 20.
Got it! I'll remember that you are allergic to peanuts.
Got it! I'll remember that you prefer italian food.
When eating out, here are some key things to be careful about:

## Food Safety
- **Cleanliness**: Check if the restaurant looks clean (tables, floors, bathrooms)
- **Food temperature**: Hot foods should be hot, cold foods should be cold
- **Proper cooking**: Ensure meat, poultry, and seafood are cooked thoroughly
- **Fresh ingredients**: Be wary if food tastes or smells off

## Health Considerations
- **Allergies**: Always inform servers about food allergies
- **Portion sizes**: Restaurant servings are often 2-3x normal portions
- **Hidden calories**: Sauces, dressings, and cooking methods can add significant calories
- **Sodium content**: Restaurant food typically contains high sodium levels

## Financial
- **Menu prices**: Check prices before ordering; ask about specials
- **Additional charges**: Service fees, tips, drinks can add up quickly
- **Specials**: Sometimes "specials" aren't actually good deals

## Practical Tips
- **Cross-contamination**: Important if you have allergies or dietary restrictions
- **Buffets**: Food sitting out for long periods can be risky
- **Raw foods**: Be cautious with raw fish, eggs, or unpasteurized items
- **Tap water safety**: In unfamiliar locations, especially when traveling

Is there a specific concern you'd like me to elaborate on?
I'd love to help you find the perfect birthday restaurant! To give you the best suggestions, could you tell me:

1. **Location**: What city or neighborhood are you in?

2. **Cuisine preference**: Any particular type of food you're craving?
   - Italian, steakhouse, seafood, Asian fusion, Mexican, French, etc.

3. **Atmosphere**: What vibe are you looking for?
   - Romantic and intimate
   - Lively and fun
   - Upscale and elegant
   - Casual and relaxed

4. **Budget**: What price range works for you?
   - $ (casual)
   - $$ (moderate)
   - $$$ (upscale)
   - $$$$ (fine dining)

5. **Party size**: Just you and one other person, or a larger group?

6. **Special requirements**: Any dietary restrictions or preferences (vegetarian, gluten-free, etc.)?

Once I know more about what you're looking for, I can suggest some great options for your special day! 🎉

Notice how the assistant:

  1. Stores facts when you use memory keywords
  2. Retrieves relevant facts when answering questions
  3. Combines stored knowledge with its language model capabilities
  4. Maintains conversation context across multiple turns

This is a complete, working memory system. But we can make it better.