Step-by-Step Problem Solving: Chain-of-Thought Reasoning for AI Agents

You've learned how to write clear prompts and use strategies like roles and examples to guide your AI agent. But what happens when you ask your agent a question that requires real thinking? Not just recalling facts, but working through a problem step by step?

Try this experiment. Ask a language model: "If a train leaves Chicago at 2 PM traveling 60 mph, and another train leaves St. Louis (300 miles away) at 3 PM traveling 75 mph toward Chicago, when do they meet?"

You might get an answer. But is it right? The model might jump straight to a conclusion without showing its work. And when the answer is wrong, you have no idea where the reasoning broke down.

Now try adding one simple phrase: "Let's think this through step by step."

Suddenly, the model shows its reasoning. It breaks down the problem, considers each piece, and works toward the answer methodically. This simple technique, called chain-of-thought reasoning, transforms how AI agents handle complex problems.

Language models are excellent at pattern matching and generating text. They can recall facts, write coherently, and follow instructions. But complex problems require more than pattern matching. They require reasoning: breaking down a problem, considering relationships, and building toward a solution.

Without explicit guidance to reason, models often take shortcuts. They might pattern-match to similar problems they've seen in training and output an answer that looks plausible but is actually wrong. This is especially common with:

• Math problems: Where each step depends on the previous one
• Logic puzzles: Where you need to track multiple constraints
• Multi-step tasks: Where you must plan a sequence of actions
• Analytical questions: Where you need to weigh evidence and draw conclusions

The solution isn't a more powerful model (though that can help). The solution is teaching the model to think through problems explicitly, showing its work as it goes.

Chain-of-thought (CoT) reasoning is simple: instead of asking the model to jump straight to an answer, you prompt it to explain its thinking step by step. You're essentially asking it to "show its work," just like a math teacher would require.

When you use chain-of-thought prompting, the model generates intermediate reasoning steps before arriving at a final answer. These steps serve two purposes:

1. They improve accuracy: By working through the problem explicitly, the model is less likely to make logical errors or skip important considerations.

2. They provide transparency: You can see how the model arrived at its answer, which helps you trust the result and debug when something goes wrong.

Think of it like the difference between asking someone "What's $17 \times 23$?" versus "What's $17 \times 23$? Show me how you calculated it." The second request produces not just an answer, but a process you can verify.

The simplest way to trigger chain-of-thought reasoning is to add a phrase like "Let's think through this step by step" or "Let's solve this step by step" to your prompt. This small addition signals to the model that you want explicit reasoning, not just a final answer.

Let's see this in action with a simple word problem:

The first response might just say "80 chairs" (which is wrong, by the way). The second response will show the reasoning:

Let's think through this step by step.

Step 1: Calculate the total number of chairs
- 23 tables $\times$ 4 chairs per table = 92 chairs

Step 2: Subtract the broken chairs
- 92 chairs - 12 broken chairs = 80 chairs

Therefore, there are 80 chairs left in the restaurant.

Let's think through this step by step.

Step 1: Calculate the total number of chairs
- 23 tables $\times$ 4 chairs per table = 92 chairs

Step 2: Subtract the broken chairs
- 92 chairs - 12 broken chairs = 80 chairs

Therefore, there are 80 chairs left in the restaurant.

Wait, that's still 80. Let me recalculate: $23 \times 4 = 92$, then $92 - 12 = 80$. Actually, that's correct! The point is that with chain-of-thought, you can verify each step. If there were an error, you'd see exactly where it occurred.

Chain-of-thought reasoning isn't necessary for every task. If you're asking for a fact ("What's the capital of France?") or a simple transformation ("Translate this to Spanish"), you don't need explicit reasoning steps. The model can answer directly.

But chain-of-thought becomes valuable when:

The problem has multiple steps: Each step builds on the previous one, and skipping a step leads to errors.

The answer isn't obvious: The problem requires analysis, not just recall.

You need to verify the logic: Being able to check the reasoning is as important as getting the right answer.

The model tends to make mistakes: If you notice the model getting certain types of questions wrong, adding chain-of-thought often helps.

Riddles are a great test case because they require careful thinking. Let's try one:

The model might respond:

Let's think through this step by step.

Step 1: Analyze the first clue - "I speak without a mouth"
This suggests something that makes sound but isn't alive or doesn't have a physical mouth.

Step 2: Consider the second clue - "hear without ears"
This means it can respond to or carry sound without having ears. This reinforces that it's not a living thing.

Step 3: Examine the third clue - "I have no body"
This confirms it's not a physical creature or person.

Step 4: Look at the final clue - "I come alive with wind"
This is the key clue. Something that needs wind to function and creates sound.

Step 5: Put it together
Something without a body that makes sound and requires wind... this describes an echo. Sound travels through air (wind), bounces off surfaces, and returns as an echo. An echo "speaks" (produces sound) without a mouth, "hears" (responds to sound) without ears, has no physical body, and exists because of sound waves traveling through air.

Answer: An echo

Let's think through this step by step.

Step 1: Analyze the first clue - "I speak without a mouth"
This suggests something that makes sound but isn't alive or doesn't have a physical mouth.

Step 2: Consider the second clue - "hear without ears"
This means it can respond to or carry sound without having ears. This reinforces that it's not a living thing.

Step 3: Examine the third clue - "I have no body"
This confirms it's not a physical creature or person.

Step 4: Look at the final clue - "I come alive with wind"
This is the key clue. Something that needs wind to function and creates sound.

Step 5: Put it together
Something without a body that makes sound and requires wind... this describes an echo. Sound travels through air (wind), bounces off surfaces, and returns as an echo. An echo "speaks" (produces sound) without a mouth, "hears" (responds to sound) without ears, has no physical body, and exists because of sound waves traveling through air.

Answer: An echo

By working through each clue systematically, the model arrives at the correct answer with clear reasoning you can follow.

Remember few-shot prompting from the previous chapter? You can combine that technique with chain-of-thought by showing the model examples of step-by-step reasoning. This is especially powerful for tasks where you want consistent reasoning patterns.

Instead of just showing input-output pairs, you show input-reasoning-output triplets. The model learns not just what to answer, but how to think about the problem.

Let's say you're building a feature where your assistant helps evaluate whether claims are supported by evidence:

The model will follow the reasoning pattern you demonstrated:

Reasoning:
- The evidence comes from a survey, which captures self-reported data
- 60% is a majority, suggesting a notable correlation
- "Felt they slept better" is subjective, not an objective measure of sleep quality
- The evidence shows correlation but doesn't prove causation (other factors could be involved)
- The sample size and methodology aren't specified, which limits confidence
Conclusion: Partially supported (shows correlation but not causation)

Reasoning:
- The evidence comes from a survey, which captures self-reported data
- 60% is a majority, suggesting a notable correlation
- "Felt they slept better" is subjective, not an objective measure of sleep quality
- The evidence shows correlation but doesn't prove causation (other factors could be involved)
- The sample size and methodology aren't specified, which limits confidence
Conclusion: Partially supported (shows correlation but not causation)

By providing examples of good reasoning, you've taught the model how to approach this type of analysis.

Let's apply chain-of-thought reasoning to make your personal assistant more capable. Here are some scenarios where it helps:

The model will break down the problem:

Let's think through this step by step.

Step 1: Calculate total time needed
- Research: 3 hours
- Create slides: 4 hours
- Practice: 2 hours
- Feedback: 1 hour
Total: 10 hours

Step 2: Calculate available time
- Thursday evening: 2 hours
- Friday evening: 2 hours
- Saturday: 6 hours
- Monday evening: 2 hours
Total: 12 hours (enough time with 2 hours buffer)

Step 3: Consider dependencies
- Must research before creating slides
- Must create slides before practicing
- Should get feedback after practicing

Step 4: Create schedule
Thursday evening (2 hours): Research (part 1)
Friday evening (2 hours): Research (part 2, complete)
Saturday (6 hours): Create slides (4 hours) + Practice (2 hours)
Monday evening (2 hours): Get colleague feedback (1 hour) + final practice (1 hour)

This schedule respects the dependencies and uses 11 of your 12 available hours.

Let's think through this step by step.

Step 1: Calculate total time needed
- Research: 3 hours
- Create slides: 4 hours
- Practice: 2 hours
- Feedback: 1 hour
Total: 10 hours

Step 2: Calculate available time
- Thursday evening: 2 hours
- Friday evening: 2 hours
- Saturday: 6 hours
- Monday evening: 2 hours
Total: 12 hours (enough time with 2 hours buffer)

Step 3: Consider dependencies
- Must research before creating slides
- Must create slides before practicing
- Should get feedback after practicing

Step 4: Create schedule
Thursday evening (2 hours): Research (part 1)
Friday evening (2 hours): Research (part 2, complete)
Saturday (6 hours): Create slides (4 hours) + Practice (2 hours)
Monday evening (2 hours): Get colleague feedback (1 hour) + final practice (1 hour)

This schedule respects the dependencies and uses 11 of your 12 available hours.

Chain-of-thought is also valuable for helping users debug their own thinking:

The model will help structure the decision:

Let's think through this step by step.

Step 1: Assess the urgency
- Current laptop still works, just slow
- This suggests want, not urgent need
- You can likely wait if needed

Step 2: Check the budget
- Total budget: \$2,000
- Laptop cost: \$1,200
- Phone cost (in 6 months): \$800
- Total needed: \$2,000
- Remaining after both: \$0

Step 3: Consider timing
- If you buy the laptop now: \$800 left for phone
- That's exactly enough for the phone
- But no buffer for unexpected expenses

Step 4: Evaluate alternatives
- Wait on laptop, buy phone when needed: keeps more flexibility
- Buy laptop now: commits most of your budget
- Fix/upgrade current laptop: might be cheaper option

Step 5: Recommendation
Given that your current laptop works (just slow) and you have a more predictable phone need in 6 months, consider:
- Option A: Wait on the laptop purchase until after you assess the phone situation
- Option B: Look into cheaper laptop upgrades (RAM, SSD) to extend current laptop life
- Option C: If the laptop significantly impacts your work/productivity, buy it now but plan carefully for the phone purchase

The decision depends on how much the slow laptop affects your daily life versus the importance of maintaining budget flexibility.

Let's think through this step by step.

Step 1: Assess the urgency
- Current laptop still works, just slow
- This suggests want, not urgent need
- You can likely wait if needed

Step 2: Check the budget
- Total budget: \$2,000
- Laptop cost: \$1,200
- Phone cost (in 6 months): \$800
- Total needed: \$2,000
- Remaining after both: \$0

Step 3: Consider timing
- If you buy the laptop now: \$800 left for phone
- That's exactly enough for the phone
- But no buffer for unexpected expenses

Step 4: Evaluate alternatives
- Wait on laptop, buy phone when needed: keeps more flexibility
- Buy laptop now: commits most of your budget
- Fix/upgrade current laptop: might be cheaper option

Step 5: Recommendation
Given that your current laptop works (just slow) and you have a more predictable phone need in 6 months, consider:
- Option A: Wait on the laptop purchase until after you assess the phone situation
- Option B: Look into cheaper laptop upgrades (RAM, SSD) to extend current laptop life
- Option C: If the laptop significantly impacts your work/productivity, buy it now but plan carefully for the phone purchase

The decision depends on how much the slow laptop affects your daily life versus the importance of maintaining budget flexibility.

This kind of structured thinking helps users make better decisions by explicitly considering all factors.

As you use chain-of-thought prompting, you'll notice certain patterns that work well for different types of problems:

For math problems: Break into calculation steps, show each intermediate result

For logic puzzles: List constraints, test possibilities, eliminate what doesn't work

For planning tasks: Identify requirements, check resources, sequence actions

For analytical questions: State the question, gather relevant facts, weigh evidence, draw conclusions

For decision-making: Define options, list pros and cons for each, compare, recommend

You don't need to specify these patterns in your prompt. Just asking for step-by-step thinking often triggers the appropriate pattern. But if the model isn't structuring its reasoning the way you want, you can provide an example that demonstrates the pattern you prefer.

Chain-of-thought reasoning is powerful, but it's not always the right tool:

It's slower: Generating reasoning steps takes more time than jumping to an answer. For simple questions, this overhead isn't worth it.

It uses more tokens: More generated text means higher API costs. Use chain-of-thought when accuracy matters more than speed or cost.

It can be verbose: Sometimes you just want a quick answer, not a detailed explanation. Match the technique to your needs.

It doesn't guarantee correctness: Chain-of-thought improves accuracy, but the model can still make errors in its reasoning. Always verify critical results.

The key is knowing when the benefits (better accuracy, transparency, debuggability) outweigh the costs (time, tokens, verbosity).

Start by applying chain-of-thought to problems where the model makes mistakes. If a simple prompt produces wrong answers, try adding "Let's think step by step." You'll quickly notice which types of problems benefit most from explicit reasoning.

Keep track of what works. When you find a prompt pattern that produces good reasoning for a particular type of problem, save it. Over time, you'll build a library of effective chain-of-thought prompts you can reuse and adapt.

Pay attention to how the model structures its reasoning. You'll start recognizing good reasoning patterns versus sloppy ones. This helps you craft better prompts and evaluate the model's outputs more effectively.

Chain-of-thought reasoning is your first tool for teaching agents to think, not just respond. By prompting the model to show its work, you get more accurate answers and insight into how it arrived at them.

But we can go further. In the next chapter, you'll learn how to make your agent check its own work and refine its answers. You'll discover techniques for getting the agent to review its reasoning, consider alternatives, and improve its responses through self-reflection. These approaches build on chain-of-thought to create even more reliable agents.

The key takeaway: when you need your agent to handle complex problems, don't just ask for an answer. Ask it to think through the problem step by step. That simple change transforms a pattern-matching system into something that can reason.

• Chain-of-thought reasoning improves accuracy by making the model show its work instead of jumping to conclusions
• The phrase "Let's think step by step" is often all you need to trigger explicit reasoning
• Use chain-of-thought for complex problems where accuracy matters more than speed
• Combine with few-shot prompting to teach specific reasoning patterns
• Verify the reasoning, not just the answer, to catch errors and build trust
• Save effective patterns to build a library of proven chain-of-thought prompts

With chain-of-thought reasoning in your toolkit, your AI agent can handle problems that require real thinking. The next chapter builds on this foundation by teaching your agent to check and refine its own reasoning.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about chain-of-thought reasoning.