Prompt Testing & Evaluation

Why Prompts Need Systematic Testing

Unlike code, prompts have no deterministic output. Testing prompts requires probabilistic approaches: run multiple times, check output patterns, compare to known-good baselines, and measure quality metrics.

Regression Tests for Prompts (pytest-based)

import pytest
from openclaw import Agent

agent = Agent(model="gpt-4o-mini", system_prompt="Classify sentiment.")

TEST_CASES = [
    ("Great product, love it!", "positive"),
    ("Broken on arrival, terrible.", "negative"),
    ("Item received.", "neutral"),
]

@pytest.mark.parametrize("text,expected", TEST_CASES)
def test_sentiment_classification(text, expected):
    result = agent.run(f"Classify: {text}").output.strip().lower()
    assert expected in result, f"Expected '{expected}' in '{result}'"

Evaluation Metrics

LLM-as-Judge Evaluation

Use a stronger model to evaluate outputs from a cheaper model:

from openclaw import Agent

judge = Agent(
    model="gpt-4o",
    system_prompt="""
Rate the following AI response on a scale of 1-5 for:
- accuracy (does it answer the question correctly?)
- format (does it follow the required format?)
- conciseness (is it appropriately brief?)
Return JSON: {"accuracy": int, "format": int, "conciseness": int, "notes": str}
"""
)

def evaluate(question: str, response: str) -> dict:
    import json
    result = judge.run(f"Question: {question}\nResponse: {response}").output
    return json.loads(result)

A/B Testing Prompt Variants

import random
from openclaw import Agent

PROMPT_A = "Summarise in 3 bullet points."
PROMPT_B = "Provide a 3-point executive summary."

def ab_run(text: str) -> tuple[str, str]:
    variant = random.choice(["A", "B"])
    prompt = PROMPT_A if variant == "A" else PROMPT_B
    agent = Agent(model="gpt-4o-mini", system_prompt=prompt)
    return variant, agent.run(text).output

Batch Evaluation With Test Datasets

import json
from pathlib import Path
from openclaw import Agent

def batch_evaluate(prompt: str, dataset_path: str) -> dict:
    agent = Agent(model="gpt-4o-mini", system_prompt=prompt)
    cases = json.loads(Path(dataset_path).read_text())
    results = {"pass": 0, "fail": 0, "errors": []}
    for case in cases:
        output = agent.run(case["input"]).output
        if case["expected"] in output:
            results["pass"] += 1
        else:
            results["fail"] += 1
            results["errors"].append({"input": case["input"], "got": output})
    return results

CI/CD Integration

Prompt tests should run automatically on every pull request that touches a .yaml promptbook file or a prompt-building function. Use GitHub Actions (or any CI system) to gate merges on passing prompt regression tests.

# .github/workflows/prompt-tests.yml
name: Prompt Tests
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'src/prompt_builder.py'

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install -r requirements-test.txt
      - run: pytest tests/prompts/ -v --tb=short
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PROMPT_VERSION: ${{ github.head_ref }}

Keep an OPENAI_API_KEY secret scoped to prompt-test runs only — never reuse your production key in CI. Use a separate sub-key with strict rate limits.

Managing Evaluation Cost

LLM calls in tests are not free. Keep costs predictable by tiering your test suite:

Smoke tests use deterministic checks (schema validation, keyword presence). Regression tests add LLM-as-judge only for critical quality dimensions. Full evals run a larger dataset with the production-quality model.

Managing Prompt Changes

Treat prompt changes like database migrations — incremental, tracked, and reversible. A prompt change log entry should include: what changed, why, measured delta in quality score, and who approved it.

# PROMPT_CHANGELOG.md
## 2026-03-15 — coding-agent v2.3.1
**Change:** Added "prefer typing.Protocol over ABC" to style guidelines.
**Reason:** 12% of code review comments requested this explicitly.
**Quality delta:** +3% on coding rubric benchmark (86 → 89).
**Approved by:** @team-ai-platform

## 2026-02-28 — coding-agent v2.3.0
**Change:** Removed "apologise if you cannot complete the task" directive.
**Reason:** Caused unnecessary apology text in outputs when tools failed.
**Quality delta:** Apology rate: 8% → 0.2%.
**Approved by:** @team-ai-platform