Monitoring — Logs, Metrics, Alerts & Tracing

Why Observability Matters

Running AI agents in production is fundamentally different from running stateless services. Each agent invocation spawns multiple LLM calls, tool executions, memory reads, and asynchronous steps — any of which can stall, fail, or produce unexpected output. Without proper observability you're debugging production incidents blind.

OpenClaw ships with built-in support for the three pillars of observability: logs capture event context and decision rationale, metrics expose quantitative health signals, and traces map the complete request flow from user input to final output. Together they give you a real-time picture of every agent run.

Enable Monitoring in 60 Seconds

Add the following block to your openclaw.yaml to activate all three observability pillars at once:

monitoring:
  enabled: true
  log_level: INFO          # DEBUG | INFO | WARNING | ERROR
  log_format: json         # json | text
  metrics:
    enabled: true
    port: 9090             # Prometheus /metrics scrape endpoint
    prefix: openclaw
  tracing:
    enabled: true
    exporter: otlp         # otlp | jaeger | zipkin
    endpoint: http://localhost:4317
    sample_rate: 1.0       # 1.0 = trace every request

The Three Pillars of AI Agent Observability

Pillar	What It Captures	Recommended Stack	Typical Retention
Logs	Agent decisions, tool calls, LLM responses, errors, context	Loki + Grafana	30–90 days
Metrics	Latency p50/p99, token budget, error rate, active agents	Prometheus + Grafana	90–365 days
Traces	End-to-end span tree, LLM call durations, tool dependencies	Tempo / Jaeger	7–30 days

Sections

📊

Monitoring Overview

Observability pillars, what to monitor, recommended stack, and quick setup.

Read →

📝

Logs

Structured JSON logging, log levels, ELK/Loki integration, and query patterns.

Read →

📈

Metrics

Prometheus metrics, Grafana dashboards, token usage, latency, and error rates.

Read →

🔔

Alerts

Alertmanager rules, PagerDuty/Slack integration, runbooks, and SLO tracking.

Read →

🔍

Distributed Tracing

OpenTelemetry integration, Jaeger/Tempo, trace agent runs, and span analysis.

Read →

Key Metrics to Track

Metric	Type	Alert Threshold
`openclaw_task_duration_seconds`	Histogram	p99 > 30s
`openclaw_token_usage_total`	Counter	Budget alert
`openclaw_tool_errors_total`	Counter	Rate > 5%
`openclaw_agent_active`	Gauge	> 100
`openclaw_llm_requests_total`	Counter	Rate limit

Grafana Dashboard Setup

Import the official OpenClaw Grafana dashboard (ID 19842) or download the JSON from the GitHub repo. It includes pre-built panels for:

Agent throughput — tasks/min over a rolling 1-hour window
LLM latency heatmap — p50 / p95 / p99 response times per model
Token budget tracker — cumulative spend vs daily budget ceiling
Tool error breakdown — error rate per tool with drill-down into stack traces via Loki
Active agents gauge — current concurrent agent count vs configured limit
Memory utilisation — vector store size, retrieval latency, eviction frequency

# Add Prometheus data source then import dashboard
curl -X POST http://admin:admin@localhost:3000/api/dashboards/import \
  -H "Content-Type: application/json" \
  -d '{ "dashboard": {"id": 19842}, "overwrite": true, "inputs": [{"name": "DS_PROMETHEUS","type": "datasource","pluginId": "prometheus","value": "Prometheus"}] }'

Structured Log Reference

When log_format: json is set, every log line is a JSON object. Common fields and their meanings:

Field	Type	Example	Description
`ts`	string	`2026-03-16T12:00:00Z`	ISO-8601 UTC timestamp
`level`	string	`INFO`	Log level
`agent_id`	string	`agnt_abc123`	Unique agent instance ID
`task_id`	string	`task_xyz`	Task correlation ID
`iteration`	int	`3`	Current think/act iteration number
`tool`	string	`web_search`	Tool being invoked (tool calls only)
`duration_ms`	int	`1420`	Duration of the current operation
`tokens_used`	int	`843`	Tokens consumed by this LLM call
`error`	string	`tool_timeout`	Error code if the step failed
`trace_id`	string	`4bf92f3577b3…`	OpenTelemetry trace ID for correlation

Prometheus Alerting Rules

Add these rules to your Alertmanager config to catch the most common production issues:

groups:
  - name: openclaw
    rules:
      - alert: HighAgentErrorRate
        expr: rate(openclaw_tool_errors_total[5m]) / rate(openclaw_llm_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "OpenClaw tool error rate > 5% for 2 minutes"

      - alert: AgentTaskTimeout
        expr: increase(openclaw_task_timeout_total[5m]) > 3
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "More than 3 agent tasks timed out in the last 5 minutes"

      - alert: TokenBudgetNear90Pct
        expr: openclaw_token_usage_total / openclaw_token_budget_total > 0.9
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "Token budget is over 90% consumed"

      - alert: LLMProviderUnreachable
        expr: openclaw_llm_provider_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "LLM provider health check failing — agents cannot execute"

OpenTelemetry Distributed Tracing

OpenClaw emits OpenTelemetry spans for every step in an agent run. Each task creates a root span with child spans for LLM calls, tool invocations, and memory operations. This lets you visualise the complete agent execution tree in Jaeger or Grafana Tempo.

# openclaw.yaml — full tracing config
monitoring:
  tracing:
    enabled: true
    exporter: otlp
    endpoint: http://otel-collector:4317  # OTLP gRPC endpoint
    sample_rate: 0.1       # sample 10% in high-volume production
    propagation: w3c       # w3c | b3 | b3multi
    resource:
      service.name: openclaw-agents
      deployment.environment: production

Useful Loki Log Queries

If you are using Grafana Loki for log aggregation, these LogQL queries are useful for debugging agent behaviour:

# All errors in the last hour
{app="openclaw"} |= "error" | json | level="ERROR"

# Slow tool calls (> 5 seconds)
{app="openclaw"} | json | duration_ms > 5000 | line_format "{{.tool}} took {{.duration_ms}}ms on task {{.task_id}}"

# Token usage per agent session (last 24h)
sum by (agent_id) (
  sum_over_time({app="openclaw"} | json | unwrap tokens_used [24h])
)

# Tasks that exceeded max_iterations
{app="openclaw"} |= "max_iterations_reached" | json

Defining SLOs for Agent Workloads

Recommended Service Level Objectives for production OpenClaw deployments:

SLO	Target	Measurement Window	Prometheus Query
Task completion rate	≥ 99.0%	30 days rolling	`1 - (rate(openclaw_task_failed_total[30d]) / rate(openclaw_task_total[30d]))`
p95 task latency	≤ 20s	1 hour rolling	`histogram_quantile(0.95, rate(openclaw_task_duration_seconds_bucket[1h]))`
API availability	≥ 99.9%	30 days rolling	`avg_over_time(openclaw_api_up[30d])`
Tool error rate	≤ 2%	5 min rolling	`rate(openclaw_tool_errors_total[5m]) / rate(openclaw_tool_calls_total[5m])`

OpenClaw AI Monitoring

Why Observability Matters

Enable Monitoring in 60 Seconds

The Three Pillars of AI Agent Observability

Sections

Monitoring Overview

Logs

Metrics

Alerts

Distributed Tracing

Key Metrics to Track

Grafana Dashboard Setup

Structured Log Reference

Prometheus Alerting Rules

OpenTelemetry Distributed Tracing

Useful Loki Log Queries

Defining SLOs for Agent Workloads