RAG Integration

How RAG Works in OpenClaw

OpenClaw's RAG pipeline has three phases:

1. Indexing: Documents are split into chunks, embedded using an embedding model, and stored in a vector database.
2. Retrieval: When the agent receives a query, it embeds the query and finds the most semantically similar chunks using vector similarity search.
3. Augmentation: Retrieved chunks are injected into the agent's context window as reference material before the LLM generates its response.

The result is an agent that can answer accurately about content it has never seen in training, using the vector search to find the most relevant passages from potentially millions of document chunks.

Setting Up RAG

# config.yaml — enable RAG
rag:
  enabled: true
  vector_store: chroma            # chroma, qdrant, pinecone, weaviate, pgvector
  collection: company_docs
  embedding_model: text-embedding-3-small
  chunk_size: 512                 # characters per chunk
  chunk_overlap: 64               # overlap between consecutive chunks
  top_k: 6                        # retrieve 6 most relevant chunks
  score_threshold: 0.72           # minimum similarity score (0-1)

  # Optional: vector store connection
  chroma:
    persist_directory: ~/.openclaw/chroma

# Index your documents
openclaw rag index --source ./docs --glob "**/*.md"
openclaw rag index --source ./wiki --glob "**/*.html" --strip-html
openclaw rag index --url https://docs.example.com --crawl-depth 3

# Verify index
openclaw rag stats
# Collection: company_docs | Documents: 1,420 | Chunks: 8,931 | Size: 42MB

# Test retrieval without running an agent
openclaw rag query "how do I configure the authentication module?"

Supported Vector Stores

For getting started, use Chroma — it requires no additional infrastructure and persists to a local directory. For production deployments, Qdrant provides excellent performance at scale and is easy to self-host on Docker or Kubernetes.

Embedding Models

The embedding model determines retrieval quality. OpenClaw supports both API-based and local embedding models:

# OpenAI embeddings (highest quality, requires API key)
embedding_model: text-embedding-3-small   # 1536 dims, $0.02/1M tokens
embedding_model: text-embedding-3-large   # 3072 dims, $0.13/1M tokens

# Local embeddings via Ollama (free, private)
embedding_model: nomic-embed-text    # via Ollama
embedding_provider: ollama
embedding_base_url: http://localhost:11434

Advanced RAG Patterns

Hybrid search: Combine vector similarity with keyword (BM25) search for better precision on technical queries where exact term matching matters alongside semantic similarity.

rag:
  retrieval_strategy: hybrid
  hybrid_alpha: 0.5    # 0.0 = pure keyword, 1.0 = pure vector, 0.5 = balanced

Parent-document retrieval: Index small chunks for precision but retrieve the full parent document for context. This balances retrieval accuracy with coherent context delivery.

rag:
  retrieval_strategy: parent_document
  child_chunk_size: 256     # small chunks for retrieval
  parent_chunk_size: 1024   # larger parent returned to LLM

Chunking Strategies

How you split documents into chunks has a major impact on retrieval quality. Fixed-size character chunking (the default) is fast and simple, but often splits in the middle of logical units like paragraphs, code blocks, or table rows, producing incoherent chunks that confuse the LLM.

OpenClaw supports several smarter chunking strategies:

• Semantic chunking: Split at sentence or paragraph boundaries, keeping semantically related content together. Better quality, slower to compute.
• Markdown-aware chunking: Split on heading boundaries (H2, H3) while preserving complete sections. Excellent for documentation sites.
• Code-aware chunking: For code repositories, split on function and class boundaries rather than character count.

rag:
  chunking:
    strategy: markdown_headers    # split on H2 headings
    fallback: sentence            # fall back to sentence boundaries
    min_chunk_size: 100           # don't create tiny chunks
    max_chunk_size: 800

Reranking for Better Retrieval

Vector similarity search retrieves candidate chunks efficiently but isn't perfectly accurate — some retrieved chunks are only superficially similar to the query. Reranking adds a second pass that uses a more accurate (but slower) cross-encoder model to reorder the top-K candidates before presenting them to the LLM.

rag:
  top_k: 20            # retrieve more candidates for reranking
  reranking:
    enabled: true
    model: cross-encoder/ms-marco-MiniLM-L-12-v2   # local cross-encoder
    top_n: 5           # return top 5 after reranking

Reranking typically improves answer quality by 10-25% on informational queries at the cost of an additional inference call. For production RAG systems where answer quality is critical, the improvement is usually worth the latency and compute cost.

Evaluating RAG Quality

RAG systems require evaluation at two levels: retrieval quality (are the right chunks being retrieved?) and answer quality (is the LLM using retrieved context correctly?). OpenClaw's RAG evaluation command runs a test set of questions with known correct answers through the full retrieve-then-generate pipeline and reports metrics.

openclaw rag eval   --test-questions tests/rag_questions.json   --metrics [retrieval_precision, answer_faithfulness, answer_relevance]

The most important metric is answer faithfulness — whether the agent's answer is supported by the retrieved context, rather than by the LLM's training knowledge. Low faithfulness indicates the LLM is hallucinating instead of using the retrieved context, often caused by poor chunk quality or a similarity score threshold that's set too low (retrieving irrelevant chunks). Increase score_threshold and improve chunking strategy to address faithfulness problems.

Managing RAG Costs

RAG adds two cost components to agent runs: embedding costs when indexing and querying, and additional context tokens fed to the LLM from retrieved chunks. Control costs by keeping top_k as small as needed — start with 4 and increase only if answer quality suffers. Use a local embedding model via Ollama for indexing large corpora, and cache embedding results for frequent queries. A caching layer for semantic search results (Redis with a 1-hour TTL is sufficient for most use cases) can reduce embedding API costs by 60-80% for agents that answer similar questions repeatedly. Also consider the score_threshold parameter — a higher threshold retrieves fewer but more relevant chunks, reducing both context tokens and noise in the LLM context.