Prompt Optimization

Token Cost vs Output Quality

Compressing System Prompts

Before (verbose, 87 tokens):

You are a helpful and knowledgeable assistant that specialises in Python programming.
When the user asks you a question about Python code, please make sure to provide a
detailed and accurate response. Always format your code with proper syntax highlighting.
Please be thorough but avoid unnecessary verbosity. Thank you!

After (concise, 31 tokens):

Python expert. Answer programming questions concisely.
Use code blocks for all code. Return only what was asked.

Rules for compression:

• Replace prose with bullet points.
• Remove filler words ("please", "make sure to", "thank you").
• Merge overlapping instructions.
• Test that quality holds after each reduction.

Structured Output to Reduce Verbosity

Prose answers are longer and harder to parse. JSON output reduces tokens and eliminates post-processing:

# Instead of asking for a paragraph explanation:
# "Explain why this code is slow and how to fix it."

# Ask for structured output:
system_prompt = """
Analyse code performance issues.
Return JSON: {"issue": str, "severity": "low"|"medium"|"high", "fix": str, "tokens_saved": int}
"""

Per-Model Optimization

Different models respond to prompts differently:

Before / After: Prompt Optimization Example

# BEFORE — 143 tokens, inconsistent output format
"""
You are an expert at summarising customer reviews. When given a customer
review, please summarise it in a way that captures the main points, the
sentiment, and any specific product features mentioned. Try to be brief.
Return your answer as a readable paragraph.
"""

# AFTER — 38 tokens, consistent JSON output
"""
Summarise customer reviews.
JSON: {"sentiment": "positive"|"negative"|"neutral", "key_points": list[str], "features": list[str]}
"""

Measuring Optimization Impact

Never ship a prompt optimization without measuring its effect on both cost and quality. An optimization that cuts token cost by 30% but drops quality by 10% may not be worth it.

import statistics, json
from openclaw import Agent

agent = Agent(model="gpt-4o-mini")

def benchmark(prompt_fn, test_cases: list[dict], judge_fn) -> dict:
    """
    prompt_fn(case) -> str: builds the prompt for a test case
    judge_fn(case, output) -> float: returns a quality score 0-1
    """
    results = []
    for case in test_cases:
        resp = agent.run(prompt_fn(case))
        results.append({
            "quality": judge_fn(case, resp.output),
            "tokens_in":  resp.usage.prompt_tokens,
            "tokens_out": resp.usage.completion_tokens,
            "cost_usd":   resp.usage.cost_usd,
        })
    return {
        "avg_quality":    statistics.mean(r["quality"] for r in results),
        "avg_tokens_in":  statistics.mean(r["tokens_in"] for r in results),
        "total_cost_usd": sum(r["cost_usd"] for r in results),
    }

Prompt Caching (OpenAI API)

OpenAI automatically caches the prefix of your prompt — the leading tokens that remain identical across requests. Prompts with a long, stable system prompt benefit the most: cache hits cost 50% less and return ~2× faster.

• Put stable content first. System prompt, tool schemas, and few-shot examples should appear before the variable user message so the maximum number of tokens are eligible for caching.
• Minimum cache length is 1,024 tokens. Short prompts (<1K tokens) are never cached; optimize those by reducing length instead.
• Cache lifetime is 5–10 minutes for most API traffic. Burst workloads with tight loops benefit most.

# Check cache hit in API response
response = client.chat.completions.create(model="gpt-4o", messages=msgs)
if hasattr(response.usage, "prompt_tokens_details"):
    cached = response.usage.prompt_tokens_details.cached_tokens
    savings = cached * 0.0000025  # gpt-4o input cache price per token
    print(f"Cache hit: {cached} tokens (≈ ${savings:.5f} saved)")

Optimization Checklist

• ✅ Use the cheapest model that passes your quality benchmark.
• ✅ Remove all filler phrases (“As an AI language model…”, “Certainly!” directives).
• ✅ Replace verbose prose instructions with structured lists or a table.
• ✅ Move static sections (examples, schema) before dynamic sections so caching applies.
• ✅ Set max_tokens to a realistic ceiling, not the API maximum.
• ✅ Use response_format: json_object to eliminate explanation preamble in schema tasks.
• ✅ Run tiktoken counts in CI to alert on unexpected prompt growth.
• ✅ A/B test every significant change against baseline before promoting.