OpenClaw AI Agent Testing

Build reliable OpenClaw agents with a comprehensive test suite: unit tests for tools, integration tests for workflows, LLM mocking, and automated CI/CD pipelines.

Testing

Why Test AI Agents?

AI agents are inherently non-deterministic — the same input can produce different outputs, tools can fail, and LLM responses change with model updates. Without a structured test suite, regressions are invisible until production breaks.

OpenClaw's testing framework gives you the tools to make agents deterministic during tests: a MockLLM replaces real API calls with scripted responses, AgentTestCase provides assertion helpers for agent runs, and the CI/CD integration runs your full suite on every pull request with zero API cost.

Testing Pyramid for AI Agents

LayerWhat to TestToolsSpeed
UnitIndividual tools, validators, pure helper functionspytest + fixturesFast (<5 s)
IntegrationMulti-step agent workflows with mocked LLMMockLLM + AgentTestCaseMedium (5–30 s)
E2EFull agent runs against real APIs in stagingReal LLM + sandbox envSlow (30 s–5 min)

GitHub Actions Quick Setup

# .github/workflows/test.yml
name: Agent Tests
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - run: pip install -e ".[dev]"
      - run: pytest tests/ -v --tb=short --cov=src
        env:
          OPENCLAW_TEST_MODE: "1"  # activates MockLLM globally

Sections

🧪

Testing Overview

Testing philosophy, test pyramid for AI agents, environment setup, and tooling.

Read →
⚗️

Unit Tests

Test individual tools, validators, and pure functions with pytest and fixtures.

Read →
🔄

Integration Tests

End-to-end agent runs, multi-step workflow testing, and real API validation.

Read →
🎭

Mocking LLMs

Mock LLM responses for deterministic tests without API calls or cost.

Read →
🚀

CI/CD Integration

GitHub Actions workflows, test matrices, coverage reports, and PR gates.

Read →

Quick Test Example

import pytest
from openclaw.testing import MockLLM, AgentTestCase

class TestMyAgent(AgentTestCase):
    def setUp(self):
        self.llm = MockLLM(responses=["Task completed successfully."])
        self.agent = create_my_agent(llm=self.llm)

    def test_basic_task(self):
        result = self.agent.run("Do a basic task")
        assert result.success is True
        assert self.llm.call_count == 1

Mocking Tools in Tests

To test an agent workflow without hitting real external APIs, replace tool implementations with mocks using the patch_tool context manager:

from openclaw.testing import patch_tool, MockLLM, AgentTestCase

class TestResearchAgent(AgentTestCase):
    def test_web_search_failure_handled_gracefully(self):
        llm = MockLLM(responses=[
            '{"thought": "I need to search", "action": "web_search", "action_input": "AI news"}',
            "I could not retrieve results. Here is what I know from memory.",
        ])
        with patch_tool("web_search", side_effect=TimeoutError("Connection timed out")):
            agent = openclaw.Agent(provider=llm, tools=["web_search"])
            result = agent.run("What are the latest AI news?")

        # Agent should recover and still produce output
        assert result.success is True
        assert "could not retrieve" in result.output.lower()

    def test_web_search_returns_mocked_data(self):
        fake_results = [{"title": "AI Regulation 2026", "url": "https://example.com", "snippet": "..."}]
        with patch_tool("web_search", return_value=fake_results):
            result = self.agent.run("Find news about AI regulation")
        assert result.success is True

Pytest Fixtures for Agents

Use fixtures to create reusable agent instances and reduce test boilerplate:

# conftest.py
import pytest
import openclaw
from openclaw.testing import MockLLM

@pytest.fixture
def mock_llm():
    """MockLLM that returns a generic success message by default."""
    return MockLLM(responses=["Task completed successfully."])

@pytest.fixture
def basic_agent(mock_llm):
    """Minimal agent for unit tests — no real API calls."""
    return openclaw.Agent(
        provider=mock_llm,
        model="mock",
        tools=["read_file", "write_file"],
        max_iterations=5,
    )

@pytest.fixture
def scripted_agent():
    """Agent with a scripted multi-turn conversation for integration tests."""
    llm = MockLLM(responses=[
        '{"thought": "I will read the file first.", "action": "read_file", "action_input": "report.txt"}',
        '{"thought": "Now I will summarize.", "action": "final_answer", "action_input": "Summary: ..."}',
    ])
    return openclaw.Agent(provider=llm, model="mock", tools=["read_file"])

End-to-End Testing Strategy

E2E tests run against real APIs in a dedicated staging environment. Key practices:

  • Separate API keys: Use a dedicated low-budget API key for CI E2E tests. Set a hard monthly spend limit to avoid runaway costs.
  • Idempotent tasks: Design E2E test tasks to be idempotent — running them twice should not corrupt state (use temp directories, unique task IDs).
  • Sandbox tools: Configure E2E tests to use sandboxed tool variants: a test-only web search API, a temporary S3 bucket, a throwaway database.
  • Run on schedule, not every push: Trigger E2E tests on schedule (nightly) or manually on release branches — not on every PR push.
  • Record results: Store E2E output artefacts (full agent logs, token counts) as CI artefacts for post-run analysis.
# tests/e2e/test_research_workflow.py
import pytest
import openclaw

@pytest.mark.e2e
@pytest.mark.skipif(not os.environ.get("RUN_E2E"), reason="E2E tests disabled in PR CI")
def test_research_and_summarise_task():
    agent = openclaw.Agent(
        provider="openai",
        model="gpt-4o-mini",   # cheapest capable model for E2E
        tools=["web_search"],
        max_iterations=10,
        timeout=120,
    )
    result = agent.run("Summarise the 3 most recent OpenAI blog posts in 2 sentences each.")
    assert result.success is True
    assert len(result.output) > 100   # sanity check — not empty
    assert result.iterations_used <= 10

Code Coverage & Quality Gates

OpenClaw enforces a minimum 80% line coverage for the src/openclaw package. Configure coverage thresholds in pyproject.toml:

[tool.pytest.ini_options]
addopts = "--cov=src/openclaw --cov-report=term-missing --cov-fail-under=80"
testpaths = ["tests"]

[tool.coverage.run]
omit = ["*/migrations/*", "*/tests/*", "*/conftest.py"]

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "raise NotImplementedError",
]

View the latest coverage badge and report at htmlcov/index.html after running pytest --cov --cov-report=html.