InteractiveChecking and Refining Agent Reasoning: Self-Verification Techniques for AI Accuracy

1Let me solve this step by step:
21. Start with 15 apples
32. Sell 8 in the morning: 15 - 8 = 7 apples left
43. Sell 6 in the afternoon: 7 - 6 = 1 apple left
5
6Answer: 1 apple remains.

1from openai import OpenAI
2
3client = OpenAI(api_key="your-api-key-here")
4
5## Initial problem solving
6problem = """A store has 15 apples. They sell 8 apples in the morning 
7and 6 apples in the afternoon. How many apples are left?"""
8
9initial_response = client.chat.completions.create(
10    model="gpt-4",
11    messages=[
12        {"role": "user", "content": f"{problem}\n\nSolve this step by step."}
13    ]
14)
15
16initial_answer = initial_response.choices[0].message.content
17print("Initial answer:", initial_answer)
18
19## Now ask the agent to verify
20verification_prompt = f"""You solved this problem:
21{problem}
22
23Your answer was:
24{initial_answer}
25
26Please verify your answer by:
271. Checking each calculation
282. Confirming the logic makes sense
293. Considering if you missed anything
30
31Are you confident in this answer? If you find any issues, provide a corrected solution."""
32
33verification_response = client.chat.completions.create(
34    model="gpt-4",
35    messages=[
36        {"role": "user", "content": verification_prompt}
37    ]
38)
39
40print("\nVerification:", verification_response.choices[0].message.content)

1Let me verify my calculations:
21. Start: 15 apples ✓
32. After morning sales: 15 - 8 = 7 ✓
43. After afternoon sales: 7 - 6 = 1 ✓
5
6Wait, let me also check if the problem is even possible:
7- Total sold: 8 + 6 = 14 apples
8- Started with: 15 apples
9- 15 - 14 = 1 apple
10
11Yes, the answer checks out. We have 1 apple remaining.

1You answered X. Are you confident in this answer? 
2Double-check your reasoning before confirming.

1You solved this using method A. Can you solve it using method B 
2and see if you get the same answer?

1Explain why you chose this approach and why your answer makes sense.

1Your answer is X. Does this satisfy all the requirements:
2- Must be less than 100
3- Must be a whole number
4- Must be positive
5
6Verify each constraint.

1from anthropic import Anthropic
2
3client = Anthropic(api_key="your-api-key-here")
4
5problem = """You have a 3-gallon jug and a 5-gallon jug. 
6How can you measure exactly 4 gallons of water?"""
7
8## Initial solution
9initial_response = client.messages.create(
10    model="claude-3-5-sonnet-20241022",
11    max_tokens=1024,
12    messages=[
13        {"role": "user", "content": f"{problem}\n\nThink through this step by step."}
14    ]
15)
16
17solution = initial_response.content[0].text
18print("Initial solution:", solution)
19
20## Verification with detailed checking
21verification = client.messages.create(
22    model="claude-3-5-sonnet-20241022",
23    max_tokens=1024,
24    messages=[
25        {"role": "user", "content": f"""You proposed this solution to the jug problem:
26{solution}
27
28Please verify by:
291. Walking through each step and tracking the water in each jug
302. Confirming you end with exactly 4 gallons
313. Checking if there's a simpler solution
32
33Show your verification work clearly."""}
34    ]
35)
36
37print("\nVerification:", verification.content[0].text)

1Here's one way to measure 4 gallons:
21. Fill the 5-gallon jug completely
32. Pour from the 5-gallon jug into the 3-gallon jug (leaving 2 gallons in the 5-gallon jug)
43. Empty the 3-gallon jug
54. Pour the 2 gallons from the 5-gallon jug into the 3-gallon jug
65. Fill the 5-gallon jug again
76. Pour from the 5-gallon jug into the 3-gallon jug until it's full (this adds 1 gallon)
87. The 5-gallon jug now contains exactly 4 gallons

1Let me verify step by step:
2
3Start: 5-gal = 0, 3-gal = 0
4
5Step 1: Fill 5-gallon jug
6→ 5-gal = 5, 3-gal = 0 ✓
7
8Step 2: Pour into 3-gallon jug
9→ 5-gal = 2, 3-gal = 3 ✓
10
11Step 3: Empty 3-gallon jug
12→ 5-gal = 2, 3-gal = 0 ✓
13
14Step 4: Pour 2 gallons into 3-gallon jug
15→ 5-gal = 0, 3-gal = 2 ✓
16
17Step 5: Fill 5-gallon jug
18→ 5-gal = 5, 3-gal = 2 ✓
19
20Step 6: Pour into 3-gallon jug (which has 2 gallons, so can take 1 more)
21→ 5-gal = 4, 3-gal = 3 ✓
22
23Result: 4 gallons in the 5-gallon jug. Solution verified!

1## Using the verification pattern from earlier
2verification_prompt = """You said the train travels 90 miles in 90 minutes.
3
4Let's verify this:
51. What is the train's speed in miles per minute?
62. How many miles would it travel in 90 minutes at that speed?
73. Does your answer match your calculation?
8
9Check your work carefully."""

1Let me recalculate:
2
31. Speed: 60 miles / 60 minutes = 1 mile per minute
42. Distance in 90 minutes: 1 mile/minute × 90 minutes = 90 miles
5
6Wait, that gives me 90 miles, which matches my answer. Let me verify once more:
7- 60 miles in 60 minutes = 1 mile per minute ✓
8- 90 minutes × 1 mile per minute = 90 miles ✓
9
10Actually, my answer was correct. The train travels 90 miles in 90 minutes.

1verification_prompt = """You said 20 machines are needed.
2
3Before confirming, let's think about this carefully:
41. How many widgets does ONE machine produce in 5 minutes?
52. How many widgets does ONE machine produce in 100 minutes?
63. How many machines do you need to produce 100 widgets in 100 minutes?
7
8Work through this step by step."""

1Let me reconsider:
2
31. If 5 machines produce 5 widgets in 5 minutes, then each machine produces 1 widget in 5 minutes.
4
52. In 100 minutes, one machine produces: 100 ÷ 5 = 20 widgets
6
73. To produce 100 widgets in 100 minutes: 100 widgets ÷ 20 widgets per machine = 5 machines
8
9Wait, I made an error. The answer is 5 machines, not 20. 
10
11Each machine produces at a rate of 1 widget per 5 minutes. In 100 minutes, each machine makes 20 widgets. So 5 machines make 100 widgets in 100 minutes.
12
13Corrected answer: 5 machines.

1def refine_answer(client, problem, max_iterations=3):
2    """Iteratively refine an answer through multiple verification passes."""
3    
4    # Initial solution
5    messages = [
6        {"role": "user", "content": f"{problem}\n\nSolve this step by step."}
7    ]
8    
9    response = client.chat.completions.create(
10        model="gpt-4",
11        messages=messages
12    )
13    
14    current_answer = response.choices[0].message.content
15    print(f"Initial answer:\n{current_answer}\n")
16    
17    # Refinement loop
18    for i in range(max_iterations):
19        messages.append({"role": "assistant", "content": current_answer})
20        messages.append({
21            "role": "user", 
22            "content": """Review your answer. Are there any errors or improvements you can make? 
23            If your answer is correct and complete, say 'VERIFIED'. 
24            Otherwise, provide an improved version."""
25        })
26        
27        response = client.chat.completions.create(
28            model="gpt-4",
29            messages=messages
30        )
31        
32        refinement = response.choices[0].message.content
33        print(f"Refinement {i+1}:\n{refinement}\n")
34        
35        if "VERIFIED" in refinement.upper():
36            print("Answer verified!")
37            break
38            
39        current_answer = refinement
40    
41    return current_answer

1from openai import OpenAI
2
3client = OpenAI(api_key="your-api-key-here")
4
5budget_problem = """I earn $4,000 per month after taxes. I want to:
6- Save 20% for retirement
7- Spend no more than 30% on rent
8- Allocate $400 for groceries
9- Set aside $200 for entertainment
10- Keep $150 for utilities
11
12How much will I have left for other expenses? Create a budget breakdown."""
13
14## Initial budget calculation
15response = client.chat.completions.create(
16    model="gpt-4",
17    messages=[
18        {"role": "user", "content": budget_problem}
19    ]
20)
21
22initial_budget = response.choices[0].message.content
23print("Initial budget:\n", initial_budget)
24
25## Verification with specific checks
26verification = client.chat.completions.create(
27    model="gpt-4",
28    messages=[
29        {"role": "user", "content": budget_problem},
30        {"role": "assistant", "content": initial_budget},
31        {"role": "user", "content": """Please verify your budget by:
321. Adding up all expenses to confirm they don't exceed $4,000
332. Checking that percentages are calculated correctly
343. Confirming the remaining amount is accurate
354. Noting if any category seems unrealistic
36
37Show your verification calculations."""}
38    ]
39)
40
41print("\nVerification:\n", verification.choices[0].message.content)

1Let me verify the budget:
2
3Income: $4,000
4
5Expenses:
6- Retirement (20%): $4,000 × 0.20 = $800 ✓
7- Rent (30%): $4,000 × 0.30 = $1,200 ✓
8- Groceries: $400 ✓
9- Entertainment: $200 ✓
10- Utilities: $150 ✓
11
12Total expenses: $800 + $1,200 + $400 + $200 + $150 = $2,750
13
14Remaining: $4,000 - $2,750 = $1,250 ✓
15
16Verification complete. The budget is mathematically correct, and you'll have $1,250 
17remaining for other expenses. All calculations check out.

1from anthropic import Anthropic
2
3client = Anthropic(api_key="your-api-key-here")
4
5problem = """A rectangular garden is 12 meters long and 8 meters wide. 
6You want to build a path 1 meter wide around the entire garden. 
7What is the area of the path?"""
8
9## Stage 1 & 2: Chain-of-thought solving
10response = client.messages.create(
11    model="claude-3-5-sonnet-20241022",
12    max_tokens=1024,
13    messages=[
14        {"role": "user", "content": f"""{problem}
15
16Think through this step by step:
171. What are the dimensions of the garden?
182. What will be the dimensions including the path?
193. How can you calculate the path area?
20
21Solve the problem showing your work."""}
22    ]
23)
24
25solution = response.content[0].text
26print("Solution:", solution)
27
28## Stage 3: Verification
29verification = client.messages.create(
30    model="claude-3-5-sonnet-20241022",
31    max_tokens=1024,
32    messages=[
33        {"role": "user", "content": f"""{problem}
34
35Your solution:
36{solution}
37
38Verify by:
391. Checking dimensions are correct
402. Confirming area calculations
413. Ensuring you calculated the path area, not the total area
42
43If you find errors, provide a corrected solution."""}
44    ]
45)
46
47print("\nVerification:", verification.content[0].text)

1class PersonalAssistant:
2    def __init__(self, client):
3        self.client = client
4        
5    def solve_with_verification(self, problem, verify=True):
6        """Solve a problem with optional verification."""
7        
8        # Initial solution
9        solution = self._solve(problem)
10        
11        if not verify:
12            return solution
13            
14        # Verification step
15        verified = self._verify(problem, solution)
16        
17        return verified
18    
19    def _solve(self, problem):
20        """Generate initial solution."""
21        response = self.client.chat.completions.create(
22            model="gpt-4",
23            messages=[
24                {"role": "user", "content": f"{problem}\n\nSolve step by step."}
25            ]
26        )
27        return response.choices[0].message.content
28    
29    def _verify(self, problem, solution):
30        """Verify and potentially refine the solution."""
31        response = self.client.chat.completions.create(
32            model="gpt-4",
33            messages=[
34                {"role": "user", "content": f"""Problem: {problem}
35
36Solution: {solution}
37
38Verify this solution. If correct, return it as-is. 
39If you find errors, return a corrected version."""}
40            ]
41        )
42        return response.choices[0].message.content