Alerts
Alerts are the active component of monitoring — they notify the on-call engineer when something needs attention. Well-designed alerts are accurate (few false positives), timely (fire before users are impacted), and actionable (the on-call engineer knows what to do when they fire).
Alerting Philosophy
Every alert should map directly to a user-visible impact (real or imminent) and should have a clear action the on-call engineer can take. An alert that fires and requires the engineer to investigate for 20 minutes before understanding what to do is a poorly designed alert — it should include a runbook link and a diagnosis summary in the alert body.
Alert fatigue is one of the most dangerous failure modes in production systems. When too many false-positive alerts fire, engineers begin ignoring their paging systems — and then miss the real incidents. Fewer, higher-quality alerts are always better than many noisy ones. Start with a small set of critical alerts and add more only as you confirm they represent real problems that need attention.
Critical Alerts — Page Immediately
| Alert | Condition | Likely Cause |
|---|---|---|
| HighTaskFailureRate | Task failure rate > 10% for 5m | LLM error, config bug, tool crash |
| LLMProviderDown | LLM error rate > 50% for 2m | Provider outage, invalid API key |
| ServiceUnhealthy | /health returns 503 for 1m | Process crash, dependency failure |
| QueueUnbounded | Queue depth > 500 for 10m | Task processing stalled, overload |
Warning Alerts — Investigate During Business Hours
| Alert | Condition | Action |
|---|---|---|
| ElevatedLatency | p95 latency > 300s for 15m | Check LLM provider status, review slow tasks |
| HighTokenUsage | Avg tokens/task increasing 25%+ week-over-week | Review recent prompt changes, check for loops |
| DiskFillRate | Disk free < 20% and decreasing | Review log rotation, semantic memory size |
Configuring Alerts
Alerts are defined in Prometheus alerting rules files and evaluated by Prometheus Alertmanager. The deploy/monitoring/prometheus/alerts/ directory contains ready-to-use alert rule files for all critical and warning alerts. Copy them to your Prometheus rules directory and reload Prometheus to activate.
PagerDuty and Slack Integration
Configure Alertmanager routing to send critical alerts to PagerDuty (for immediate paging) and warning alerts to a Slack channel (for daytime review). The Alertmanager config in deploy/monitoring/alertmanager.yml includes both routing configurations with placeholders for your PagerDuty integration key and Slack webhook URL.
Runbook Linking
Every critical alert should link to a runbook with step-by-step diagnosis and remediation instructions. Store runbooks in the project wiki or a private documentation platform. Include the runbook URL in the alert's annotations field — Alertmanager includes annotation values in notification messages, making the runbook immediately accessible when the alert fires at 3am and the on-call engineer needs to act fast.
Alert Silencing and Maintenance Windows
During planned maintenance (upgrades, restarts, database migrations), alerts that fire due to the maintenance are noise that suppress the signal of real problems. Configure Alertmanager silences before beginning maintenance — set the silence to last 15 minutes longer than the expected maintenance window as a buffer. After maintenance, remove the silence and verify that alerts are back in their normal state.
Never silence alerts for longer than necessary. A silence that was set for a 2-hour maintenance window and left running for 3 days means your alerting was effectively disabled for 3 days — any production incident during that period would have gone undetected. Set calendar reminders to remove silences if you can't commit to removing them immediately after maintenance ends.
Alert Tuning Over Time
Alert rules require ongoing tuning as your traffic patterns and system behavior evolve. An alert threshold that was appropriate at 100 tasks/day may produce constant false positives at 10,000 tasks/day due to higher natural variance. Review alert thresholds quarterly, comparing alert history against actual incidents. Alerts that fired but didn't correspond to user-visible problems should be tightened or converted to warnings. Alerts that should have fired but didn't should be reviewed for threshold or query issues.
Document every alert threshold change with a comment in the alert rule file explaining the reasoning: why was the threshold changed, what data informed the decision, and when should it be reviewed again. This narrative documentation prevents alert rules from becoming arbitrary magic numbers that nobody understands.
Escalation Policy
Critical alerts that aren't acknowledged within 15 minutes should escalate automatically to a secondary on-call engineer. Configure Alertmanager's repeat interval and escalation routes to implement this. The escalation policy should be documented in the team runbook so all engineers know what to expect when they're on call: who gets paged first, after how many minutes does it escalate, and to whom.
Review and test your escalation policy occasionally — confirm that the escalation contacts are current, that paging integrations (PagerDuty, Opsgenie) are correctly connected, and that all on-call engineers have push notifications enabled on their phones. A monitoring system that can't reach the on-call engineer is only marginally better than no monitoring at all.
Alert Documentation
Every alert rule should include annotations that describe what the alert means and what to do. Prometheus alert annotations support templating — include the actual current values in the annotation text so the on-call engineer sees "Task failure rate is 23.4% (threshold: 10%)" rather than just "Task failure rate is high."
Maintain a runbook wiki page for each critical alert. The runbook should answer: what does this alert mean, what impact is the user experiencing, what is the most likely cause, how do I diagnose which cause it is, and what is the remediation for each cause. A complete runbook turns a stressful 3am incident into a repeatable procedure.
Testing Your Alerts
An untested alert is an unknown — it may fire when it should, or it may silently fail when you need it most due to a misconfigured query, a broken Alertmanager routing rule, or an expired PagerDuty API key. Test critical alerts regularly: use Prometheus's /api/v1/alerts test endpoint to manually trigger alerts and verify that notifications are delivered correctly. Do this as part of every infrastructure change and quarterly as a regular health check of your monitoring stack.