Agent Type

Research Agent with OpenClaw

A research agent searches the web, fetches and reads source pages, deduplicates information across sources, extracts citations, and produces structured Markdown reports — automating hours of manual research in minutes.

Research Agent Architecture

The research agent follows a gather-synthesise-report loop:

  1. Search: Generate 3–5 targeted queries for the research topic.
  2. Fetch: Retrieve full content from the top results for each query.
  3. Deduplicate: Merge overlapping sources, track unique citations.
  4. Synthesise: Extract key claims and supporting evidence per source.
  5. Report: Write a structured Markdown report with citations.

Required Tools

ToolPurpose
web_searchDuckDuckGo / Google SERP API search
fetch_urlDownload and extract text from URLs
text_summarizeCondense long pages to key points
file_writerSave the final report to disk

Basic Research Agent Setup

# agents/researcher.yaml
name: researcher
model: gpt-4o
temperature: 0.4
max_tokens: 4096
tools:
  - web_search
  - fetch_url
  - text_summarize
  - file_writer
system: |
  You are an expert researcher. When given a topic:
  1. Generate 4 specific search queries.
  2. Search and fetch the top 3 results per query.
  3. Summarise each source in 200 words.
  4. Write a structured report:
     ## Summary (executive overview)
     ## Key Findings (bullet list with citations [1], [2]...)
     ## Sources (numbered list with URL + title)
  Always cite your sources. Never hallucinate facts.
from openclaw import Agent

agent = Agent.from_config("agents/researcher.yaml")
report = agent.run("Research the current state of AI agent frameworks in 2025")
print(report)

Multi-Source Research with Deduplication

from openclaw import Agent
from urllib.parse import urlparse

def research_with_dedup(topic: str, max_sources: int = 10) -> str:
    agent = Agent.from_config("agents/researcher.yaml")
    seen_domains: set[str] = set()

    # Ask agent to return structured JSON of sources first
    sources_json = agent.run(
        f"Find the top web sources for: {topic}. "
        f"Return JSON array: [{{'url': '...', 'title': '...'}}]"
    )
    import json, re
    # Extract JSON from possible markdown fences
    m = re.search(r'\[.*\]', sources_json, re.DOTALL)
    sources = json.loads(m.group()) if m else []

    unique_sources = []
    for s in sources:
        domain = urlparse(s["url"]).netloc
        if domain not in seen_domains and len(unique_sources) < max_sources:
            seen_domains.add(domain)
            unique_sources.append(s)

    urls = "\n".join(f"- {s['url']}" for s in unique_sources)
    return agent.run(f"Research '{topic}' using ONLY these sources:\n{urls}")

Competitor Research Agent

# agents/competitor-researcher.yaml
name: competitor-researcher
model: gpt-4o
temperature: 0.2
max_tokens: 6144
tools:
  - web_search
  - fetch_url
  - file_writer
system: |
  You are a competitive intelligence analyst.
  When given a competitor name:
  1. Find their pricing page, feature list, and recent blog posts.
  2. Extract: pricing tiers, key features, positioning, target audience.
  3. Compare against OpenClaw's strengths and weaknesses.
  4. Output a structured competitive profile in Markdown.
  Be factual. Only use information found in the sources.
agent = Agent.from_config("agents/competitor-researcher.yaml")
profile = agent.run("Analyse LangChain as a competitor to OpenClaw")
with open("reports/langchain-competitor-profile.md", "w") as f:
    f.write(profile)

Rate Limiting and Polite Scraping

# agents/researcher.yaml — add rate limiting config
tools:
  fetch_url:
    delay_between_requests: 2.0   # seconds between fetches
    max_requests_per_domain: 3    # don't hammer a single site
    respect_robots_txt: true
    timeout: 15
    user_agent: "OpenClaw Research Agent / https://openclaw-wiki.top"
Ethical scraping: Always set a descriptive user_agent, respect robots.txt, and add delays. Some sites block scrapers — if a URL fails, the agent should gracefully skip it and note the failure in the report.

Report Generation (Markdown + PDF)

import subprocess, os
from openclaw import Agent

def generate_pdf_report(topic: str, output_dir: str = "reports") -> str:
    os.makedirs(output_dir, exist_ok=True)
    agent = Agent.from_config("agents/researcher.yaml")
    md_content = agent.run(f"Research: {topic}")

    slug = topic.lower().replace(" ", "-")[:40]
    md_path = f"{output_dir}/{slug}.md"
    pdf_path = f"{output_dir}/{slug}.pdf"

    with open(md_path, "w") as f:
        f.write(md_content)

    # Convert to PDF using pandoc (optional, if installed)
    result = subprocess.run(
        ["pandoc", md_path, "-o", pdf_path, "--pdf-engine=xelatex"],
        capture_output=True
    )
    if result.returncode == 0:
        return pdf_path
    return md_path  # fallback: return MD path if pandoc not available