Testing Overview

Why Testing Matters

AI agent systems present unique testing challenges. Unlike deterministic software where the same inputs always produce the same outputs, LLM-driven agents are probabilistic — the same task can produce different (and valid) outputs on each run. Tests must accommodate this while still catching genuine regressions.

OpenClaw's answer is a three-layer strategy: unit tests that mock LLM responses to test deterministic logic reliably; integration tests that call real LLMs but assert structural properties (tool was called, output contains key facts) rather than exact text; and behavioral regression tests that baseline agent output quality against a reference prompt set and flag large deviations for manual review.

Types of Tests in OpenClaw

The test suite is organized by category:

• Unit tests (tests/unit/): Test individual functions and classes with all external dependencies mocked. Run in milliseconds, no API keys needed. These are the primary regression-catching tests.
• Integration tests (tests/integration/): Test component interactions with real LLM calls. Require OPENAI_API_KEY. Assert structural properties (did the agent call the tool, did it finish in under 60s) rather than exact output content.
• E2E agent tests (tests/e2e/): Full agent runs against real tasks with reference output comparison. Used to catch behavioral regressions when prompt templates or tool implementations change.
• Plugin tests (tests/plugins/): Test custom tool implementations in isolation using the ToolTestHarness class.

Testing Architecture

All tests use pytest as the test runner and pytest-asyncio for async test support. Mock LLM responses are provided by openclaw.testing.MockLLMProvider, which lets you script exact sequences of LLM responses to drive deterministic test scenarios. The mock provider records all calls made to it, so tests can also assert on what the LLM was asked (verifying that the system prompt was assembled correctly, that tool results were included in context, etc.).

The openclaw.testing.AgentTestCase base class provides helper methods for common testing patterns: running an agent with a scripted LLM, asserting that specific tools were called, checking that outputs match patterns, and validating final state after a multi-turn interaction.

Test Configuration

Test behavior is controlled via environment variables and a pytest.ini config file. The most important settings:

Running the Test Suite

# Run all unit tests (no API key needed)
pytest tests/unit/ -v

# Run with coverage report
pytest tests/unit/ --cov=openclaw --cov-report=html

# Run integration tests (requires OPENAI_API_KEY)
pytest tests/integration/ -v --timeout=120

# Run a specific test file
pytest tests/unit/test_tool_registry.py -v

# Run tests matching a keyword
pytest -k "test_memory" -v

Coverage Goals

OpenClaw targets 80% line coverage for all core library code. Coverage is measured per-commit in CI and reported as a comment on pull requests. Coverage drops below 80% will block a PR from merging. New features are expected to include tests with their initial PR — test-free PRs will be returned for tests before review.

Coverage of agent behavior (which is inherently probabilistic) is measured differently: the E2E test suite defines a set of canonical tasks that the agent must complete successfully, and the pass rate across 10 runs must be above 90%. If a change drops the pass rate below that threshold, it indicates a meaningful behavioral regression.

Continuous Integration

The CI pipeline runs on every pull request and on every merge to main. The unit test job runs in under 2 minutes; integration tests run in about 10 minutes. The E2E behavioral regression tests run nightly against main, not on every PR, to avoid burning unnecessary API credits. Any nightly failure notifies the maintainer channel in Discord immediately.

Testing Philosophy: Confidence Over Perfection

Good tests don't aim for 100% certainty — they aim for an appropriate level of confidence that the system works correctly for the most important use cases, at a cost proportional to that confidence. Spending 40 hours writing tests that catch 0.1% more bugs is usually a worse investment than spending those 40 hours improving documentation, fixing known bugs, or improving the user experience.

The right testing strategy asks: what are the highest-consequence failures? What user actions are most common? Where do regressions most often appear? These questions guide where to invest in tests. The answer for OpenClaw is: the agent ReAct loop, the tool calling mechanism, the memory assembly pipeline, and the configuration validation logic. These components are tested exhaustively. Peripheral utilities get lighter coverage.

Writing Testable Agent Configurations

The discipline of writing testable code extends to agent configurations. A testable agent configuration is one that produces consistent, verifiable behavior given consistent inputs. Several patterns improve testability:

Use deterministic seeding: Set temperature: 0 in test configurations to reduce output variance. While temperature-zero outputs aren't byte-for-byte identical across model versions, they are much more consistent than higher-temperature outputs, making structural assertions more reliable.

Keep system prompts focused: A system prompt that clearly specifies the agent's role, permitted tools, and output format makes it much easier to write assertions about expected behavior. An agent with a vague system prompt can behave in many valid ways, making it hard to write meaningful tests.

Separate concerns: Agents that do one thing well are easier to test than agents that do many things. A research agent and a writing agent as separate pipeline stages are individually much more testable than a monolithic "research and write" agent.

Debugging Test Failures

When a test fails, systematic debugging is faster than guessing. Start by running the failing test in verbose mode to see the full failure output: pytest tests/ -v -s --tb=long. The -s flag disables output capture, showing all print statements and log output from the test run. The --tb=long flag shows the full traceback including local variables at each frame.

For agent test failures specifically, enable debug logging to see the full request-response sequence: set OPENCLAW_LOG_LEVEL=DEBUG before running tests. The debug log shows every message sent to the LLM, every tool call and response, and every memory retrieval operation. This is usually enough to identify exactly where the behavior diverged from expectation.