Agent Type

Web Browsing Agent with OpenClaw

A web agent extends OpenClaw with the ability to browse the internet: fetch static pages, interact with JavaScript-heavy SPAs via Playwright, fill forms, click buttons, take screenshots, and perform authenticated scraping — all controlled by natural language instructions.

Web Agent Capabilities

  • Fetch and extract text from any public URL
  • Navigate multi-page sites and follow links
  • Interact with SPAs via Playwright (click, type, scroll)
  • Fill and submit forms programmatically
  • Perform authenticated scraping (login flows)
  • Take screenshots and extract structured data from screenshots
  • Monitor pages for changes

requests/httpx vs Playwright: When to Use Each

Criterionrequests/httpxPlaywright
Page renders via JavaScriptNoYes
Need to click/scroll/interactNoYes
SpeedVery fastSlow (~3–10s per page)
Resource usageMinimalHigh (full browser)
Login / session cookiesManualBuilt-in context
Start simple: Use fetch_url (httpx) first. Only reach for Playwright when the page requires JavaScript rendering or interaction.

Simple URL Fetching and Extraction

# agents/web-agent.yaml
name: web-agent
model: gpt-4o-mini
temperature: 0.0
tools:
  - web_search
  - fetch_url
system: |
  You browse the web to answer questions.
  Always cite the URL you got information from.
  If a page is unavailable, try an alternative source.
from openclaw import Agent

agent = Agent.from_config("agents/web-agent.yaml")
result = agent.run("What is the current price of Bitcoin? Check coinmarketcap.com")
print(result)

Interactive Browser Agent (Playwright)

pip install playwright
playwright install chromium
from playwright.sync_api import sync_playwright
from openclaw import Agent

agent = Agent.from_config("agents/web-agent.yaml")

def browse_with_playwright(url: str, task: str) -> str:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(
            user_agent="Mozilla/5.0 (compatible; OpenClaw-Bot/1.0)"
        )
        page = context.new_page()
        page.goto(url, wait_until="networkidle", timeout=30_000)
        
        # Get full page text
        content = page.evaluate("() => document.body.innerText")
        browser.close()
    
    return agent.run(f"From this page content, {task}:\n\n{content[:8000]}")

result = browse_with_playwright(
    "https://pypi.org/project/openclaw/",
    "Extract the latest version number and release date"
)
print(result)

Form-Filling Agent

from playwright.sync_api import sync_playwright

def fill_contact_form(url: str, data: dict) -> bool:
    """Fill and submit a contact form on a webpage."""
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        page.goto(url)
        
        # Fill fields by label text (more resilient than CSS selectors)
        if "name" in data:
            page.get_by_label("Name").fill(data["name"])
        if "email" in data:
            page.get_by_label("Email").fill(data["email"])
        if "message" in data:
            page.get_by_label("Message").fill(data["message"])
        
        # Take screenshot before submission for audit trail
        page.screenshot(path="form-before-submit.png")
        
        # Submit (comment out for dry-run)
        # page.get_by_role("button", name="Submit").click()
        # page.wait_for_url("**/thank-you**", timeout=10_000)
        
        browser.close()
        return True

Frequently Asked Questions

How do I handle CAPTCHA challenges with web agents?

For production, use 2Captcha or Anti-Captcha APIs: the agent detects a CAPTCHA, sends the challenge to the service, receives the solution, and submits it. This adds 10–30 seconds per solve and costs $1–$3 per 1000 CAPTCHAs. For internal tools, use authenticated sessions (login once, reuse cookies) to avoid CAPTCHAs entirely. Some sites use hCaptcha/reCAPTCHA v3 that score behavior rather than requiring user interaction — slow down click timing to mimic humans.

Can web agents work on JavaScript-heavy single-page apps?

Yes, but use Playwright or Selenium instead of simple HTTP requests. These tools run a real Chromium browser that executes JavaScript, waits for DOM updates, and handles dynamic content. Set appropriate page.wait_for_selector() calls to ensure elements are loaded before interaction. Be aware that browser automation is slower (~2–5 seconds per page) and more resource-intensive than raw HTTP scraping.

How do I keep web agents working when sites change their HTML structure?

Use semantic selectors instead of brittle CSS class names: Playwright's page.get_by_role("button", name="Submit") is more resilient than .btn-primary-submit-v2. Store selectors in a config file, not hardcoded in agent logic, so you can update them without redeploying. Add automated tests that run daily against target sites and alert when selectors break. For high-value scraping, consider paid APIs (Product Hunt API, Crunchbase API) instead of raw scraping.

What are the legal risks of web scraping agents?

Scraping public data is generally legal (US: hiQ v. LinkedIn precedent), but violating Terms of Service can lead to account bans or legal action. Always check robots.txt and respect crawl-delay directives. Avoid scraping personal data (GDPR/CCPA risk). For commercial use, prefer official APIs or licensed data providers. Add rate limiting (max 1 request/sec) and a descriptive User-Agent with contact info (MyBot/1.0 (contact@example.com)) to demonstrate good faith.

Anti-Bot Detection Considerations

Always check the Terms of Service before scraping a site. Many sites explicitly prohibit automated access. Respect robots.txt, use realistic delays, and set a descriptive User-Agent header. 
import random, time

def polite_fetch(urls: list[str]) -> list[str]:
    """Fetch multiple URLs with random delays to avoid rate limits."""
    agent = Agent.from_config("agents/web-agent.yaml")
    results = []
    for url in urls:
        result = agent.run(f"Fetch and summarise: {url}")
        results.append(result)
        # Random delay between 2 and 5 seconds
        time.sleep(random.uniform(2.0, 5.0))
    return results