Distributed Tracing

What is Distributed Tracing

A trace is a tree of spans, where each span represents one unit of work with a start time, end time, parent span, and any number of attributes and events. For an OpenClaw task, the root span is the overall task execution; child spans cover each reasoning step; grandchild spans cover each LLM call and tool invocation within those steps. Viewing the complete trace as a Gantt chart reveals immediately whether time is dominated by LLM API latency, tool execution, or agent reasoning.

OpenTelemetry Integration

OpenClaw uses the OpenTelemetry SDK for tracing. Enable it with three environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_SERVICE_NAME=openclaw
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1   # sample 10% of traces

Compatible with any OpenTelemetry-compliant backend: Jaeger, Tempo, Zipkin, Datadog, Honeycomb, and others. No code changes required — instrumentation is automatic through the OpenClaw OTEL middleware.

Trace Structure

A typical task trace contains: one root span (task.execute); N reasoning step spans (agent.step); within each step, one LLM call span (llm.call) and zero or more tool spans (tool.{name}); and optionally memory retrieval spans (memory.retrieve). Span attributes include model name, token counts, tool arguments, and error details. This structure makes it easy to answer questions like "which tool consumed the most time in this task?" and "how many LLM calls did this task make?"

Sampling Strategy

At high task throughput, collecting every trace is prohibitively expensive. Use head-based sampling (decide at trace start whether to sample, based on a percentage) for low-traffic services. For high-traffic services, use tail-based sampling (collect all traces, decide at trace completion whether to keep them) to retain 100% of slow or error traces for investigation while discarding routine fast-and-successful traces.

Trace Storage and Viewing

Grafana Tempo is the recommended trace backend for teams already using the Prometheus/Grafana stack — it integrates directly with Grafana for trace search and provides a Grafana panel that correlates traces with logs and metrics for the same time range. Jaeger is the traditional open-source choice with a richer trace-centric UI. Both are available as Docker Compose services in deploy/monitoring/.

Using Traces for Debugging

When a task takes unexpectedly long or fails, querying by task_id in the trace UI immediately shows the full execution graph. You can see: which step took longest, whether the LLM was slow or whether a tool was slow, whether any span produced an error, and what the LLM was given in context for each call. This level of detail resolves in 5 minutes what might take hours of log analysis to reconstruct from plain-text log files.

Trace Context Propagation

When OpenClaw tasks are triggered by external services (webhooks, API calls from your application), trace context propagation allows the external service's trace to be extended into the OpenClaw execution. Pass the traceparent and tracestate W3C Trace Context headers in the task submission request, and OpenClaw will use the external trace's context as the parent for the task's root span.

This means you can trace a user request from your frontend, through your application backend, into OpenClaw's agent execution, into the LLM API call, as a single connected trace. No more manually correlating separate traces with task IDs — the trace itself shows the complete causal chain from user action to LLM response.

Trace Analysis Patterns

Three analysis patterns that deliver the most value from traces:

Slow task investigation: Query for traces with root span duration above the p95 threshold. Examine the waterfall to identify whether slowness is in LLM API calls (provider issue) or tool execution (tool optimization needed) or agent reasoning loop count (prompt inefficiency). Each cause has a different remediation.

Error investigation: Query for error traces by filtering on spans with error=true. The error span shows exactly which step failed, with the error message and any attributes that identify the root cause. For LLM errors, the LLM call span includes the HTTP status and response body; for tool errors, the tool span includes the tool's exception details.

Performance trending: Compare p95 latency distributions across traces from this week vs. last week. A shift in the distribution without a corresponding shift in task complexity indicates a regression — in LLM provider performance, in a tool implementation, or in memory retrieval efficiency.

Instrumentation Best Practices for Plugins

When building custom tools or plugins, add manual instrumentation to provide visibility into your code's behavior. Use the OpenTelemetry API to create child spans around significant operations — database queries, external HTTP calls, file I/O. Spans created inside a tool's execution automatically become children of the tool's span in the trace, giving complete visibility without any additional trace context management.

from opentelemetry import trace

tracer = trace.get_tracer("my_plugin")

async def my_tool(query: str) -> dict:
    with tracer.start_as_current_span("my_tool.db_query") as span:
        span.set_attribute("db.query", query[:100])  # truncate long queries
        results = await run_db_query(query)
        span.set_attribute("db.result_count", len(results))
    return {"results": results}

Keep span attribute values short and structured. Long text values bloat trace storage and make the trace UI harder to use. Use attributes for categorization and quantification (counts, IDs, status codes) and use span events for timestamped narrative description of what happened within the span.

Traces and Cost Analysis

Traces are the most detailed tool for understanding why a specific task was expensive. When a task consumes far more tokens than expected, view its trace and examine the LLM call spans — specifically the token_usage.total attribute on each LLM call span. If one step consumed 80% of the tokens, view the span's input context attribute to understand why the context was so large. Common causes: retrieved memory chunks that are too long, unbounded tool output being passed back to the LLM, or very long working memory from a deep conversation history.