Implementing Memory in Our Agent › Design Decisions and Trade-offs

Let's discuss the choices we made and their implications:

We limit conversation history to 20 messages. This:

Pros:

• Keeps costs predictable
• Prevents context overflow
• Simple to implement

Cons:

• Loses older conversation context
• Might forget important details from earlier in the session

Alternative: Use summarization instead of hard truncation. Periodically summarize old messages and keep the summary.

We detect "remember", "store", etc. to trigger memory storage. This:

Pros:

• Simple and fast
• No extra API calls
• User has explicit control

Cons:

• Might miss implicit memory requests
• Requires specific keywords

Alternative: Use the LLM to classify every message as "store this" or "just chat". More flexible but costs more.

We use a local embedding model for semantic search. This:

Pros:

• Fast and free
• Good enough for most use cases
• No API calls for every search

Cons:

• Less powerful than API-based embeddings
• Requires installing additional libraries

Alternative: Use OpenAI's embedding API or Anthropic's future embedding service. Better quality but costs money.

We store facts in a JSON file. This:

Pros:

• Simple to implement
• No database setup required
• Easy to inspect and debug

Cons:

• Doesn't scale to thousands of facts
• No concurrent access support
• Limited query capabilities

Alternative: Use a proper database (SQLite for local, PostgreSQL for production) or a vector database (Pinecone, Weaviate, Chroma).