Chain-of-Thought Reasoning in AI — How Agents Think Step by Step

What Is Chain-of-Thought?

CoT was introduced in Wei et al. (2022) "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." The core finding: models are far more accurate on reasoning tasks when prompted to show intermediate reasoning steps, compared to prompting for the answer directly.

Example without CoT:

Q: A store sells 3 items for $4 each and 2 items for $7 each. Total?
A: $26

Example with CoT:

Q: A store sells 3 items for $4 each and 2 items for $7 each. Total?
A: Let me work through this step by step.
   3 items × $4 = $12
   2 items × $7 = $14
   Total = $12 + $14 = $26

Zero-Shot CoT

Simply adding "Let's think step by step." to a prompt activates CoT reasoning without any examples. This works surprisingly well on large models (GPT-4, Claude 3).

from openclaw import Agent

agent = Agent(
    model="gpt-4o",
    system_prompt="You are a helpful assistant. Think step by step before answering.",
)
result = agent.run("If a train travels 150km in 90 minutes, what is its speed in km/h?")

Few-Shot CoT

Provide 2–3 worked examples in the system prompt or few-shot examples block. The model mirrors the reasoning style.

few_shot_examples = """
Example:
Q: Sam has 4 apples. He gives away 2 and buys 5 more. How many?
A: Start with 4. Subtract 2 gives 2. Add 5 gives 7. Answer: 7.

Example:
Q: A rectangle is 8m wide and 6m tall. What is its area?
A: Area = width × height = 8 × 6 = 48m². Answer: 48m².
"""

agent = Agent(
    model="gpt-4o-mini",
    system_prompt=f"Solve math problems step by step.\n{few_shot_examples}",
)

When CoT Helps Most

Task type	CoT benefit
Multi-step arithmetic	Very high — reduces calculation errors
Logical reasoning	High — forces consistent reasoning chains
Code debugging	High — surfaces assumptions
Entity extraction	Low — direct answer is usually better
Sentiment classification	Low — reasoning can overcomplicate
Simple factual retrieval	None — adds unnecessary tokens

Self-Consistency

Self-consistency (Wang et al. 2022) extends CoT by sampling multiple reasoning chains with temperature > 0, then taking a majority vote over the final answers. This improves accuracy on difficult reasoning tasks at the cost of 3–10× more tokens.

planning:
  strategy: "cot"
  self_consistency:
    enabled: true
    samples: 5          # Generate 5 independent reasoning chains
    temperature: 0.7    # Diversity in reasoning paths
    voting: "majority"  # majority | weighted | best_of_n

Configuring CoT in OpenClaw

from openclaw import Agent

agent = Agent(
    model="gpt-4o",
    reasoning={
        "mode": "cot",              # cot | auto | off
        "show_in_output": False,    # Hide reasoning from final answer
        "prefix": "Let me think carefully about this.\n\n",
    },
)

Setting show_in_output: False strips the reasoning trace from the user-facing response while keeping it in the context for subsequent steps.

Key takeaways:

CoT consistently improves accuracy on multi-step reasoning, logic, and arithmetic tasks.
"Let's think step by step" is a free performance boost for large models.
Use few-shot CoT examples when format consistency matters.
Self-consistency (5 samples + majority vote) is the highest-accuracy CoT variant.
Avoid CoT for simple classification or retrieval tasks — it adds cost with no benefit.

Chain-of-Thought Prompt Templates

The phrasing of your CoT instruction significantly affects output quality. These templates consistently produce strong reasoning traces:

Style	Template	Best for
Standard CoT	"Think step by step, then answer."	General reasoning
Plan-first	"First, outline a plan. Then execute each step."	Multi-step tasks
Question decomposition	"Break this into sub-questions. Answer each, then synthesise."	Research, analysis
Devil's advocate	"Consider the strongest counter-argument before concluding."	Decision-making
Show your work	"Show all calculations and cite your sources."	Maths, fact-checking

Cost Considerations

Chain-of-thought prompting increases output token count by 2–5× per query. At scale, this has significant cost implications. Optimise by:

Using a cheaper model (GPT-4o-mini, Claude Haiku) for the reasoning step and a stronger model only for the final answer synthesis
Caching CoT outputs for identical or near-identical inputs using semantic caching
Disabling CoT for high-volume, low-complexity queries (classification, routing) and reserving it for queries where reasoning quality matters