Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Short-Term Conversation Memory › The Context Window Challenge

Michael BrenndoerferJuly 1, 20251 min read

AI Agent HandbookAll chapters

Part 6 of 15Memory and Retrieval - How Agents Remember

Chapter 15Short-Term Conversation Memory › The Context Window Challenge

Reading section 8 of 15 · The Context Window Challenge · 1 minSection 8 of 15 · 1 min

Section 8 of 15 · 1 min read

Short-Term Conversation Memory › The Context Window ChallengeOne section · Sections in this chapter8 / 15 ↓

Sections in this chapter15

Here's a problem: conversation history grows with every exchange. After 50 turns, you're sending 100 messages (50 user, 50 assistant) with every new question. This creates two issues:

Cost: Most API providers charge per token. Sending the entire history every time gets expensive.

Context limits: Models have maximum context windows. Claude Sonnet 4.5 supports 200K tokens, but you'll eventually hit limits in very long conversations.

Let's see this problem in practice:

In[12]:

Code

## After a long conversation
print(f"Messages in history: {len(agent.conversation_history)}")
## Output: 156 messages

## Estimate token count (rough approximation)
total_chars = sum(len(str(msg)) for msg in agent.conversation_history)
estimated_tokens = total_chars // 4  # Rough estimate: 1 token ≈ 4 characters
print(f"Estimated tokens: {estimated_tokens}")
## Output: Estimated tokens: 12,450

Out[12]:

Console

Messages in history: 12
Estimated tokens: 341

For a long conversation, you might be sending thousands of tokens with each request. Most of those tokens are old messages that might not be relevant anymore.