Maintenance and Updates › Updating Prompts and Instructions
AI Agent HandbookAll chapters Maintenance and Updates › Updating Prompts and InstructionsOne section · Sections in this chapter9 / 19 ↓
One of the most common maintenance tasks is refining the prompts and instructions you give your agent. As you learn how users interact with the agent and what works well, you'll want to adjust these prompts.
When to Update Prompts
Consider updating prompts when:
- Users frequently misunderstand the agent's responses
- The agent frequently misunderstands user requests
- You add new tools and need to teach the agent when to use them
- You discover edge cases the current prompt doesn't handle well
- A new model version interprets prompts differently
How to Update Prompts Safely
Prompt changes can have surprising effects. A small wording change might significantly alter the agent's behavior. So treat prompt updates like code updates: test thoroughly before deploying.
Here's a process:
1. Document the Current Prompt: Before changing anything, save the current version. You might need to revert.
2. Make the Change: Update the prompt in your development environment.
3. Test with Examples: Run your test cases. Compare responses from the old and new prompts. Are the changes what you expected?
4. Test Edge Cases: Try unusual inputs. Does the new prompt handle them better or worse?
5. A/B Test (Optional): If possible, run both versions in production for a while, randomly assigning users to each. Compare metrics to see which performs better.
6. Deploy Gradually: Deploy the new prompt to a small percentage of users first. Monitor for issues. If it looks good, gradually increase the percentage until everyone uses the new version.
Example: Refining Tool Usage
Let's say your agent has a calculator tool, but it sometimes tries to do math in its head instead of using the tool. You want to update the prompt to encourage tool usage.
Current prompt:
system_prompt = """
You are a helpful assistant. You have access to a calculator tool for math.
Answer user questions accurately and helpfully.
"""Updated prompt:
system_prompt = """
You are a helpful assistant. When users ask math questions, always use the calculator tool
rather than attempting to calculate in your head. This ensures accuracy.
For other questions, respond based on your knowledge.
"""Test this with various math questions. Does the agent now consistently use the calculator? Does it still handle non-math questions well? If yes, deploy it.