# Runbooks

Tested response procedures for the alerts auto-provisioned by
[`stats-lake/observability/`](https://github.com/decocms/stats-lake/blob/main/observability/README.md).

Every alert generated by `provision-dashboards.ts` carries a
`runbook_url` annotation that points to a Markdown file in this
directory. Each runbook follows the same structure so on-call can read
top-to-bottom under stress:

1. **What this alert means.** One paragraph, no jargon.
2. **First check (60 seconds).** The single most likely cause and how to
   confirm it.
3. **Diagnostic queries.** ClickHouse SQL the responder can paste into
   Grafana / ClickStack to dig deeper.
4. **Common causes & fixes.** Ranked by frequency. Each with a "did
   that fix it?" verification step.
5. **Escalation.** When to page a domain owner, and who.
6. **Post-mortem hook.** What to capture so the post-mortem isn't
   reconstructed from memory.

## Runbook catalogue

| Alert ID                  | Runbook                                                |
|---------------------------|--------------------------------------------------------|
| `http-error-spike`        | [`http-error-spike.md`](./http-error-spike.md)         |
| `http-latency-spike`      | [`http-latency-spike.md`](./http-latency-spike.md)     |
| `cache-hit-drop`          | [`cache-hit-drop.md`](./cache-hit-drop.md)             |
| `commerce-upstream-slow`  | [`commerce-upstream-slow.md`](./commerce-upstream-slow.md) |
| `tail-exception-spike`    | [`tail-exception-spike.md`](./tail-exception-spike.md) |

## Authoring a new runbook

When you add a new alert template, add a runbook with the same ID. The
provisioning script doesn't enforce the link today — that gate lands in
Phase 6 (Governance), but treat it as required: an alert without a
runbook is a half-shipped alert.

Keep runbooks short. If a section grows past 20 lines, split it out
into a dedicated incident-pattern doc and link to it from the runbook.
