Monitoring Overview

Running an AI agent in production without monitoring is running blind. You don't know if tasks are completing, if LLM latency is degrading, if error rates are creeping up, or if the system is about to exhaust its budget — until users start complaining. Monitoring gives you visibility so you can catch and fix problems before they become incidents.

Why Monitor an AI Agent

AI agent systems have properties that make monitoring more challenging than traditional software. Output quality is subjective and hard to measure automatically. LLM latency is high and variable — a "slow" response that would be unacceptable for a web request is normal for an agent that does 5 minutes of research. Cost is a runtime concern, not just a development cost.

Despite these challenges, monitoring is essential. Production incidents in unmonitored systems are discovered by users, not engineers — and by then the damage (lost tasks, wasted tokens, degraded user experience) has already happened. A good monitoring setup surfaces problems in seconds or minutes, not hours or days.

What to Monitor

For OpenClaw in production, the critical monitoring signals are:

  • Task success rate: What percentage of submitted tasks complete successfully? A success rate below 95% is a warning sign; below 90% is a production incident.
  • Task latency (p50/p95/p99): How long do tasks take? Track percentiles, not just averages. A rising p99 with stable p50 indicates occasional very slow tasks that may be hitting timeouts.
  • LLM API error rate: Rate limit errors, authentication errors, and service unavailability from the LLM provider. These are external and sudden — you need immediate alerting if they spike.
  • Token usage per task: Rising token usage over time indicates prompt bloat, memory issues, or looping behavior. Set an alert when average token use increases more than 25% week-over-week.
  • Queue depth: How many tasks are waiting to run? A growing queue indicates the system can't keep up with submission rate.
  • Memory utilization: RAM consumption for semantic memory indexes grows with usage. Monitor it to prevent OOM crashes.

Monitoring Stack

OpenClaw emits Prometheus-format metrics from the /metrics endpoint. The recommended monitoring stack is Prometheus for metric collection, Grafana for dashboards and alerting, and Loki (or Elasticsearch) for log aggregation. This stack is available as a Docker Compose file in deploy/monitoring/ for quick setup.

# Start the full monitoring stack
cd deploy/monitoring
docker compose up -d

# Access Grafana at http://localhost:3000 (default: admin/openclaw)
# Pre-built dashboards are imported automatically from provisioning/

Dashboards and Alerting

The provided Grafana dashboards cover four views: the executive overview (task count, success rate, cost), the operational overview (latency percentiles, queue depth, error rate), the LLM provider view (per-provider latency and error rates), and the resource view (CPU, memory, disk for the OpenClaw process). Each dashboard includes alert rules for critical thresholds — import them from deploy/monitoring/grafana/alerts/.

Log Aggregation

OpenClaw writes structured JSON logs that Loki or Elasticsearch can parse and index efficiently. Configure your log shipper (Promtail for Loki, Filebeat for Elasticsearch) to read from the log file or container stdout. Key log fields to index are: task_id, level, event, agent_name, tool_name, duration_ms, and error_code. With these indexed, you can instantly query all logs for a specific failing task or all invocations of a specific tool.

Health Checks

The GET /health endpoint returns the current health status of the OpenClaw process and its dependencies. Configure your load balancer and Kubernetes probes to call this endpoint. A 200 OK response indicates the system is healthy; a 503 indicates a dependency is unavailable (LLM provider unreachable, database down, etc.). The response body includes which health check failed, enabling faster root-cause diagnosis.

On-Call and Incident Response

Define an on-call rotation before deploying to production, not after the first incident. The on-call engineer should have access to Grafana dashboards, application logs, and the ability to restart the service and roll back versions. Keep a runbook documenting common failure modes and their remediation. A runbook for "LLM API rate limit exceeded" that explains how to identify the cause, how to drain the queue gracefully, and how to contact the LLM provider is worth hours when an actual incident hits at 2am.

Monitoring in Practice: Getting Started

Most teams are tempted to skip monitoring until they have an incident. This is backwards — configure basic monitoring before your first production deployment, not after your first outage. The minimum viable monitoring setup for OpenClaw takes under an hour to configure and immediately gives you the visibility needed to operate confidently.

Start with three things: (1) Set up the /health endpoint as a health check in your load balancer or Kubernetes deployment — this is often built into infrastructure and costs nothing to add. (2) Enable structured JSON logging and forward logs to any log aggregation service (even a simple Elasticsearch + Kibana setup). (3) Scrape the /metrics endpoint with Prometheus and set up at least the task failure rate alert. These three steps take you from blind to informed.

After your first week in production, review what questions you found yourself trying to answer that your monitoring couldn't answer — those gaps identify the next monitoring improvements. Iterate from there, adding dashboards and alerts as you learn what matters for your specific workload.

Cost Observability

Token usage is real money in production. OpenClaw tracks token consumption per task and emits it as a metric, but connecting those numbers to dollars requires knowing the cost per token for your model. Configure the cost per 1K tokens for your model in the OpenClaw settings, and the system will compute estimated cost per task and emit it as a metric. This enables Grafana dashboards showing daily/weekly cost trends and alerts when costs spike unexpectedly.

Set monthly cost budgets and alert when you reach 80% of budget — this gives you time to investigate and adjust before you overspend. Typical causes of cost spikes: a looping agent that makes hundreds of LLM calls on a single task, a data pipeline with unbounded input that feeds extremely long contexts, or a sudden increase in task submission volume. All three are diagnose-able from metrics in minutes when you have cost monitoring in place.