OpenClaw Performance Benchmarks

Benchmark Methodology

All benchmarks were run on a standard EC2 t3.medium instance (2 vCPU, 4GB RAM) with OpenClaw v1.5.2. Each test ran 100 iterations; results show median (P50) and P95 latencies. LLM calls used the provider's public API with US-East endpoints.

Latency Benchmarks by Model

Simple Task (single LLM call, no tools)

Complex Task (3 tool calls + reasoning)

Throughput Benchmarks

Concurrent task processing capacity with different max_workers settings on a t3.large (2 vCPU, 8GB RAM):

Diminishing returns appear above 8 workers — the bottleneck shifts from CPU to LLM API rate limits.

Impact of Response Caching

Benchmark: 1,000 tasks from a real-world FAQ-answering workload (typical 40% cache hit rate):

Semantic caching (matching similar but not identical queries) achieves the best results at 55% cost reduction. See Response Caching for setup.

Token Usage Benchmarks

Token costs are often more impactful than latency. Here's how different context management strategies compare:

Run Your Own Benchmarks

OpenClaw includes a benchmark runner. Create a benchmark file:

# benchmark.yaml
name: "My Benchmark Suite"
tasks:
  - prompt: "Summarize the key benefits of Python decorators"
    expected_keywords: ["wrapper", "function", "decorator"]
  - prompt: "List 5 uses of Redis in production"
    expected_keywords: ["cache", "queue", "session"]
runs: 50
models: [gpt-4o-mini, claude-3-haiku]
compare_profiles: [dev, production]

openclaw benchmark run benchmark.yaml --output results.json
openclaw benchmark report results.json --format html

The HTML report shows latency histograms, cost breakdowns, and quality scores side by side.

Benchmark Methodology

Meaningful benchmarks measure what your production workload actually does, not what a synthetic test does. Before running any benchmark, define the specific question you're trying to answer: "How many tasks per minute can this agent process with this model?" or "What is the p95 latency for a typical file analysis task?" A well-defined question determines which metrics to collect, what workload to simulate, and what constitutes a satisfactory result. Tests without a specific question tend to produce data that looks impressive but doesn't guide actionable decisions.

Use production-representative inputs for your benchmark workload. Synthetic inputs (all the same length, all the same structure) produce misleading results because the model's behavior varies significantly with input characteristics. Sample a hundred representative tasks from your production logs and use those as your benchmark input corpus. This ensures your benchmark captures the real variance in your workload — some tasks will be fast, some slow, and the distribution will match what users actually experience.

Key Metrics to Track

Track throughput (tasks per minute), latency at p50/p95/p99 percentiles, token consumption per task, tool call count per task, and error rate. Don't optimize for just one metric — a configuration that maximizes throughput by reducing model quality is counterproductive. Track all metrics together and look for trade-off points: where does increasing max_tokens improve output quality without significantly hurting throughput? Where does adding another tool call improve results without meaningfully increasing latency? These trade-off insights guide configuration decisions grounded in data rather than intuition.

Run benchmarks in conditions that match production as closely as possible: same network environment, same hardware tier, same model, same time of day (API rate limits and response times can vary). A benchmark run on a developer's laptop at night produces different results than production traffic on cloud infrastructure during peak hours. Establish your benchmark environment as close to production as feasible and run all comparison tests in the same environment.

Interpreting Results

Statistical significance matters for benchmark results. A single run may show a 5% improvement that is pure measurement noise. Run each configuration at least 5 times and compare the distributions rather than individual runs. If the 95% confidence intervals of two configurations overlap, the difference is not statistically significant — you haven't demonstrated a real improvement. Report median and p95 latency rather than mean — outlier-heavy distributions make the mean misleading, while median and p95 tell you what typical and worst-case behavior look like.

Benchmark Tooling

Use openclaw bench --runs 10 --task-file benchmark-inputs.jsonl --output results.csv to run a standardized benchmark suite. The --runs flag controls how many times each input is evaluated — more runs increase statistical confidence but cost more. Export to CSV for analysis in a spreadsheet or import into a monitoring dashboard. Store benchmark results alongside your git commits to track performance over time and quickly identify the commit that caused a regression.