Logs

Log Format

Every log line is a JSON object with a consistent set of base fields:

{
  "timestamp": "2024-10-01T14:23:45.123Z",
  "level": "INFO",
  "event": "task.completed",
  "task_id": "task_abc123",
  "agent_name": "research_agent",
  "duration_ms": 18432,
  "token_usage": {"prompt": 2341, "completion": 891, "total": 3232},
  "tool_calls": 3
}

Domain-specific events include additional fields. Tool execution events include tool_name, tool_args (sanitized of sensitive values), and tool_result_length. Error events include error_code, error_message, and traceback. The consistent base fields allow all events to be correlated by task_id and filtered by level or event.

Log Levels

Key Log Events

The events most useful for operational monitoring are:

• task.submitted — A task was accepted into the queue
• task.started — Task execution began
• task.completed — Task finished successfully with output and token counts
• task.failed — Task failed; includes error code and final state
• tool.called — A tool was invoked; includes tool name and argument summary
• llm.request — An LLM API call was made; includes model, token counts, latency
• llm.error — LLM API returned an error; includes HTTP status and error message
• budget.exceeded — Task exceeded its configured token or time budget

Structured Log Queries

With structured JSON logging, log queries become powerful and precise. In Loki (LogQL), you can query: {app="openclaw"} | json | event="task.failed" | error_code != "budget_exceeded" — showing all genuine failures excluding budget overruns. In Elasticsearch (Lucene): event:task.failed AND NOT error_code:budget_exceeded AND duration_ms:>60000 — long-running failures. Build saved searches for your most common operational queries to reduce investigation time during incidents.

Sensitive Data in Logs

Logs must never contain sensitive data: API keys, user credentials, personally identifiable information, or the full content of user messages (which may contain private data). OpenClaw's log sanitization applies automatically to tool arguments and LLM request bodies — fields matching common sensitive patterns (key, secret, password, token, authorization) are replaced with [REDACTED]. Review your tool implementations to verify that tool outputs, which are not automatically sanitized, don't include sensitive fields.

Log Retention

Balance retention duration against storage costs and compliance requirements. For most deployments, INFO logs from the past 30 days are sufficient for operational debugging. ERROR and CRITICAL logs should be retained for at least 90 days. Compliance-sensitive deployments may require longer retention — consult your legal or compliance team before setting retention policies. Implement log rotation (Logrotate on Linux, logging handlers in Python) to prevent log files from filling the disk on long-running deployments.

Log Shipping Configuration

OpenClaw writes logs to stdout and optionally to a file (configurable via logging.file_path in the config). The most common log shipping configurations:

Docker with Loki: Configure Docker's logging driver to use loki-docker-driver — logs from all containers are automatically shipped to Loki with container metadata as labels. No additional configuration in OpenClaw needed.

Kubernetes with Fluent Bit: Deploy Fluent Bit as a DaemonSet with the Kubernetes filter plugin to add pod metadata (namespace, pod name, container name) to all log records. The Loki output plugin streams to your Loki instance. Use the Parser filter with the json parser to extract structured fields from OpenClaw's JSON logs into Loki labels.

Bare metal with Filebeat: Configure Filebeat to tail the OpenClaw log file, parse JSON, and ship to Elasticsearch. Use an index template in Elasticsearch that maps OpenClaw-specific fields (task_id, duration_ms, token_usage) to their correct types for efficient querying.

Log-Based Alerting

Not all important events are captured by metrics. Log-based alerting catches patterns in log content that don't have dedicated metrics. Configure log alerts for: any occurrence of level: CRITICAL (immediate page), more than 5 occurrences of event: task.failed in a 5-minute window, any occurrence of event: budget.exceeded with token_usage.total > 50000 (runaway task), and any occurrence of error_code: rate_limit_exceeded from the LLM provider.

Grafana Loki supports alerting on log patterns via Grafana Alerting — define a query that counts matching log lines and alert when the count exceeds a threshold. This approach has a few minutes latency compared to Prometheus metric alerts but works for events that aren't tracked as metrics.

Debugging a Specific Task with Logs

When a user reports that a specific task produced incorrect output or failed, the task ID is your starting point. Every log line for that task includes the task_id field. Query for all logs with that task ID to see the complete execution history: submission, each reasoning step, each LLM call, each tool call, and the final completion or failure event.

The sequence of events tells a clear story. If the last event before failure was tool.called for a specific tool, the tool likely threw an exception. If the last event was llm.request with a long elapsed time before an llm.error, the LLM provider timed out. If there were 20 agent.step events before a budget.exceeded, the agent looped without making progress — review the tool results to understand why the agent didn't complete.

Log Format Customization

OpenClaw's default structured JSON log format is optimized for machine parsing, but human readability during development is also important. Set logging.format: console in your development config to get colored, human-friendly log output. The console format shows timestamp, level, event name, and key-value pairs in a compact single-line format that is easy to read while watching logs during development.