Web Scraper Agent

What We'll Build

A job listing scraper that: 1) receives a list of job board URLs, 2) reads each page, 3) extracts title, company, location, and salary, 4) deduplicates, 5) saves a jobs.csv.

Agent Configuration

# scraper-agent.yaml
name: job-scraper
description: "Scrapes job listings and extracts structured data"

llm:
  model: gpt-4o-mini
  temperature: 0

system_prompt: |
  You are a data extraction agent. For each URL provided:
  1. Read the page content
  2. Extract ALL job listings as a JSON array
  3. Each item must have: title, company, location, salary (null if not shown), url
  4. Return ONLY the JSON array — no other text

tools:
  - read_url
  - file_write

output:
  schema:
    type: array
    items:
      type: object
      required: [title, company, location, url]
      properties:
        title: {type: string}
        company: {type: string}
        location: {type: string}
        salary: {type: ["string", "null"]}
        url: {type: string}

Pipeline Configuration

# scrape-pipeline.yaml
name: job-scrape-pipeline
steps:
  - id: scrape_each
    type: for_each
    iterate_over: "{{ input.urls }}"
    as: url
    step:
      tool: read_url
      params:
        url: "{{ url }}"
    collect_as: raw_pages

  - id: extract_jobs
    type: for_each
    iterate_over: "{{ raw_pages }}"
    as: page
    step:
      tool: llm_extract
      params:
        content: "{{ page.text }}"
        schema: "{{ agents.job_scraper.output.schema }}"
    collect_as: job_lists

  - id: flatten_and_dedupe
    type: transform
    input: "{{ job_lists }}"
    operations:
      - flatten: true
      - unique_by: "title+company"
    output_as: unique_jobs

  - id: save_csv
    tool: file_write
    params:
      path: ./jobs.csv
      format: csv
      data: "{{ unique_jobs }}"

Running the Scraper

openclaw pipeline run scrape-pipeline.yaml \
  --input '{"urls": [
    "https://example.com/jobs/python",
    "https://example.com/jobs/ai-ml"
  ]}'

# Watch progress
openclaw logs --follow

# View result
head -n 5 jobs.csv

Handling JavaScript-Rendered Pages

For sites that require JavaScript execution (SPAs), use the read_url_js tool instead of read_url. This invokes a headless Playwright browser:

# Install Playwright support
pip install openclaw[playwright]
playwright install chromium

tools:
  - read_url_js   # Uses Playwright headless browser
  - file_write

What This Agent Does

This tutorial builds a web scraper agent that can visit URLs, extract structured information from web pages, and compile results into structured output. Unlike a traditional scraper (which requires you to write CSS selectors or XPath queries for each site), this agent uses the LLM to understand page content and extract the information you want in plain English. The result is a scraper that works across different page layouts without per-site configuration.

Step 1 — Agent Configuration

Create web-scraper-agent.yaml:

agent:
  name: web-scraper
  description: Extracts structured information from web pages
  system_prompt: |
    You are a web scraping specialist. When given a URL, you visit
    the page, read its content, and extract the specific information
    requested by the user. Format extracted data clearly, using
    tables or structured lists when the data has consistent structure.
    Always note the URL you scraped and the date/time of extraction.

llm:
  model: claude-3-5-sonnet-20241022
  temperature: 0.1
  max_tokens: 4096

tools:
  - name: web_fetch
    enabled: true
    config:
      timeout: 30
      max_content_length: 50000
      respect_robots_txt: true

The low temperature (0.1) is intentional — for data extraction tasks you want consistent, predictable output rather than creative variation. The respect_robots_txt: true setting ensures the agent respects site crawling policies.

Step 2 — Run a Basic Scrape

Try extracting information from a public page:

openclaw run --config web-scraper-agent.yaml   --task "Go to https://example.com and extract: page title, main heading, and paragraph count"

The agent fetches the page, reads its full content, and extracts the requested fields without any CSS selectors or scraping rules. The LLM understands the page structure semantically.

Step 3 — Batch Scraping Multiple URLs

Create a task file scrape-task.txt:

Scrape the following URLs and extract the product name, price, and availability from each:
- https://example.com/product/1
- https://example.com/product/2
- https://example.com/product/3

Return the results as a markdown table with columns: URL, Product Name, Price, Available.

Run the batch task:

openclaw run --config web-scraper-agent.yaml --task-file scrape-task.txt --output results.md

The agent works through each URL sequentially, extracts the data, and compiles the results into the requested table format saved to results.md.

Step 4 — Add Output Validation

For production scraping, add structured output validation to ensure extracted data matches your expected schema. Update the system prompt to include output format requirements:

  system_prompt: |
    You are a web scraping specialist. Extract the requested data and
    ALWAYS return it as valid JSON matching this schema:
    {
      "source_url": "string",
      "extracted_at": "ISO 8601 timestamp",
      "data": { ... extracted fields ... },
      "errors": [] or ["list of any extraction issues"]
    }
    If a field cannot be found, set it to null and add a note to errors.

Step 5 — Schedule Regular Scrapes

For monitoring tasks (price tracking, content change detection), schedule the agent to run automatically. OpenClaw integrates with system schedulers — add a cron entry for Linux/macOS:

# Run every hour and append results to a log file
0 * * * * cd /path/to/project && openclaw run   --config web-scraper-agent.yaml   --task-file scrape-task.txt   --output-append results-log.jsonl

Ethical and Legal Considerations

Always check a site's robots.txt and terms of service before scraping. The respect_robots_txt: true config handles the technical enforcement, but the legal and ethical compliance is your responsibility. Add delays between requests for high-volume scraping using the request_delay config option to avoid overloading servers. Never scrape personal data without appropriate legal basis — most privacy laws impose strict requirements on automated collection of personal information.

Troubleshooting

If the agent returns incomplete data, the page may require JavaScript rendering — add use_browser: true to the tool config to enable headless browser mode. If you hit rate limits on the target site, increase request_delay. If the LLM truncates output for large pages, increase max_tokens or reduce max_content_length to feed the LLM a smaller portion of the page.

Handling Dynamic Content and Anti-Bot Measures

Many modern sites rely heavily on JavaScript to render content, meaning a simple HTTP fetch returns only a bare HTML shell with little usable content. Enable headless browser mode with use_browser: true in the web_fetch tool config to handle these sites. The trade-off is speed — browser rendering takes 3-10x longer than a raw fetch. Use browser mode selectively by listing specific domains that require it rather than enabling it globally.

Some sites implement anti-bot measures that block automated requests. Respect these — they exist for legitimate reasons including preventing server overload and protecting proprietary data. If a site blocks your requests despite respectful scraping behavior, consider whether there's an official API, an RSS feed, or a data export function that provides the same information with explicit permission to use it programmatically. Using official APIs is always preferable to scraping when one exists.

Structuring Extracted Data for Downstream Use

Raw agent output is useful for one-off queries, but for recurring scraping tasks you want consistent structured output that downstream processes can parse reliably. Instruct the agent to produce JSON output conforming to a specific schema, then validate that schema before saving. If the agent's output doesn't conform (the LLM occasionally reformats values), the validation step catches it early rather than letting malformed data propagate to downstream consumers.

Define your output schema in the system prompt with an example. Show the exact JSON structure you want, including the data types for each field and what should be null vs omitted when a field isn't available. Concrete examples in the prompt dramatically reduce format variation in the output — the model has a clear target to hit rather than interpreting a verbal description of the desired structure.

Performance Optimization for Large-Scale Scraping

For scraping hundreds or thousands of URLs, sequential processing becomes slow. Use OpenClaw's batch processing mode to process multiple URLs in parallel: define your URL list as a structured input file and use --batch-file urls.json with a parallelism setting. Balance parallelism against the target site's rate limits — too many concurrent requests risks getting your IP blocked, while too few wastes time. A concurrency of 3-5 requests is a reasonable starting point for most sites. Always test at low concurrency first and only increase it after verifying the target handles it gracefully.

Next Steps

Extend your scraper agent by combining it with a database writer tool to persist results across runs, enabling change detection by comparing new scrapes against stored baselines. Pair it with a notification tool to send alerts when monitored pages change in ways that matter — a price drop, a new blog post, a status page update. The scraper agent is a building block; most production scraping workflows combine it with storage, diffing, and notification components to create a complete monitoring pipeline rather than a standalone tool.