Integration Tests

Integration Test Philosophy

Integration tests are necessarily slower and more expensive than unit tests — they make real API calls, take real time, and incur real costs. The discipline is to write them only where they add irreplaceable value: testing interactions with external services, verifying end-to-end behavior of complex component chains, and confirming that agent configurations produce coherent output for representative real tasks.

Integration tests assert on structural properties and invariants, not on exact output text. A test like "the agent returned a response containing at least 100 words and called the web_search tool at least once" is a good integration test. A test like "the agent returned exactly 'Here are the search results:'" is a bad integration test that will fail every time the LLM's phrasing changes.

Setting Up for Integration Tests

Integration tests require API credentials. Configure them in your environment before running:

# Required for integration tests
export OPENAI_API_KEY=sk-...
export OPENCLAW_TEST_MODEL=gpt-4o-mini   # use cheap model for tests

# Optional: also test Anthropic provider
export ANTHROPIC_API_KEY=sk-ant-...

# Run integration tests
pytest tests/integration/ -v --timeout=180

# Skip expensive tests (marked with @pytest.mark.expensive)
pytest tests/integration/ -v -m "not expensive"

Use gpt-4o-mini or claude-haiku for integration tests — they are significantly cheaper than larger models while still being capable enough for the test tasks. Reserve powerful models for E2E behavioral regression tests where quality matters.

Writing Integration Tests

Integration tests follow the same pytest structure as unit tests, but use the real Agent class with a real LLM provider:

import pytest
from openclaw import Agent

@pytest.mark.integration
@pytest.mark.asyncio
async def test_agent_completes_simple_task():
    agent = Agent.from_config("tests/fixtures/basic_agent.yaml")
    result = await agent.run_async("What is 15 * 23?")
    assert result.status == "completed"
    assert "345" in result.output
    assert result.tool_calls == []   # arithmetic should not need tools

Fixture Recording and Playback

For integration tests that don't actually need to verify live LLM behavior (just the interaction structure), use the record/playback system. Set OPENCLAW_RECORD_MODE=true to run the test against a real LLM and record all API responses to a fixture file. Future test runs with OPENCLAW_PLAYBACK_MODE=true replay the recorded responses, making the test deterministic and free. This is the recommended approach for API integration tests that can't use mocks.

LLM Provider Matrix Testing

When a new LLM provider is added, a provider matrix test verifies that all standard agent capabilities work with that provider: basic task completion, tool calling, streaming output, context window handling, and error recovery. The matrix test is parameterized over all configured providers, so adding a new provider automatically includes it in the test run.

Test Data Management

Integration test fixtures (recorded LLM responses, sample agent configs, reference output files) live in tests/fixtures/. Fixtures are committed to the repository so CI can run playback-mode integration tests without API keys. When underlying behavior changes and recorded fixtures become stale, re-record them with OPENCLAW_RECORD_MODE=true, review the diff in the recorded responses, and commit the updated fixtures with a clear commit message explaining what changed.

Managing Integration Test Costs

Integration tests that call real LLM APIs cost money. The cumulative cost of poorly designed integration tests — tests that run on every commit, use expensive models, and make many unnecessary API calls — can consume a meaningful budget over months of active development. Managing this cost proactively is part of responsible test suite maintenance.

Use the cheapest capable model for integration tests. gpt-4o-mini costs about 30x less than gpt-4o and is perfectly adequate for testing structural properties like "did the agent call the tool" or "did the agent return a complete sentence." Reserve expensive models for E2E quality tests where frontier capability genuinely matters.

Run integration tests on PR only when relevant files change — use paths filters in GitHub Actions workflows to skip integration tests on commits that only touch documentation, tests, or unrelated utilities. This can reduce integration test runs by 70%+ on projects with active documentation.

Writing Assertions for Probabilistic Outputs

The core challenge of integration testing LLM-driven agents is that outputs are probabilistic. You cannot assert on exact output text without creating a test that fails on every minor LLM update. The solution is to assert on structure, invariants, and facts rather than phrasing.

Instead of: assert result.output == "The capital of France is Paris." — which will fail when the LLM rephrases — write: assert "Paris" in result.output and "capital" in result.output.lower(). Instead of asserting on the number of sentences, assert on a minimum word count. Instead of asserting on specific tool arguments, assert on the presence of key information in those arguments: assert "France" in tool_call.args["query"].

Testing Multi-Agent Pipelines

Pipeline integration tests verify that stages pass data correctly between agents and that the final output reflects the work of all stages. Test pipelines with a minimal but real example: a two-stage pipeline (research → summarize) that runs in under 30 seconds is sufficient to verify the pipeline orchestration, inter-stage context passing, and output serialization. Full production-scale pipelines with 6+ stages are better tested in E2E tests run less frequently.

Test Cleanup and Isolation

Integration tests that leave state behind can cause test order dependencies — test B passes when run alone but fails when run after test A because test A's leftover state interferes. Always clean up after integration tests: delete temporary files, roll back database transactions, reset configuration overrides, and close any open resources.

Use pytest fixtures with yield for setup-teardown patterns. The code before yield runs as setup; the code after yield runs as teardown even if the test fails. This is more reliable than try/finally blocks inside tests and keeps the test code focused on assertions rather than cleanup.