Speeding Up the Agent › Optimizing Prompt Size
Michael BrenndoerferAugust 24, 20251 min read
AI Agent HandbookAll chapters Part 15 of 15Performance and Cost Optimization - Doing More with Less
Speeding Up the Agent › Optimizing Prompt SizeOne section · Sections in this chapter9 / 15 ↓
Sections in this chapter15
Large prompts take longer to process. Every token in your prompt adds a small amount of latency. If you're including long system messages, conversation history, or retrieved documents, consider trimming them.
Here's a practical approach:
In[9]:
Codeimport os
from anthropic import Anthropic
client = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
def trim_conversation_history(messages, max_messages=5):
"""Keep only the most recent messages to reduce prompt size."""
if len(messages) <= max_messages:
return messages
# Keep the most recent exchanges
return messages[-max_messages:]
def summarize_long_context(text, max_length=500):
"""Truncate very long context to essential information."""
if len(text) <= max_length:
return text
# Simple truncation (in production, you might use a summary model)
return text[:max_length] + "... [truncated]"
## Example usage
conversation_history = [
{"role": "user", "content": "Tell me about Python"},
{"role": "assistant", "content": "Python is a programming language..."},
# ... many more messages ...
]
## Trim history before sending
trimmed = trim_conversation_history(conversation_history, max_messages=5)
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=512,
messages=trimmed
)By keeping only the five most recent messages, you reduce the prompt size and speed up processing. The agent loses some context, but for many conversations, recent messages are all that matter.