Incident Runbooks That Work Under Pressure
PagerDuty goes off. Database connection errors on the checkout service. Your on-call engineer, bleary-eyed, opens the runbook wiki page. Step 1: “Check if the database is healthy.” No command. No expected output. No guidance on what “healthy” means when connections are failing but the instance is technically running. She opens a terminal and starts guessing. psql -h prod-db.internal -c "SELECT 1" times out. Is that the database or the network? She checks the VPC flow logs. Twenty minutes later, she discovers the connection pool is saturated, not the database itself. The fix takes 30 seconds: restart PgBouncer. The investigation took over twenty minutes because the runbook described intentions instead of actions.
The aircraft manual was on the bookshelf. The pilot needed the checklist strapped to her knee. “Check if engine is healthy” is not a checklist item. “Read gauge 3A. If below 40 PSI, execute procedure 7” is.
Audit runbooks across your org and you’ll see this pattern everywhere. The NIST Computer Security Incident Handling Guide (SP 800-61) defines the process framework, but the gap between what teams write during a calm documentation sprint and what survives a P1 is huge. Authors evaluate completeness. On-call engineers evaluate executability, three Slack threads deep, VP asking for updates every two minutes.
- Executable steps include the command, expected output, and decision tree for each outcome. Anything less is a wish list, not a runbook. An aircraft manual, not a checklist.
- Runbooks should be code, not wiki pages. Executable scripts with human checkpoints where the engineer decides the next action based on system-presented results.
- DORA’s research confirms MTTR is the reliability metric most correlated with team effectiveness. Automated diagnostics shift the engineer’s job from investigating to deciding.
- Runbook rot is the default state. Infrastructure changes. The runbook doesn’t. A checklist for a 737 being used on a 787. Half the switches moved. Validate every 90 days or after any architecture change.
- Post-mortem action items are where improvements go to die. Enforce a 48-hour SLA for runbook updates identified during incidents. Track closure rate as a team metric.
What Separates a Runbook from Documentation
| Wiki Documentation | Executable Runbook | |
|---|---|---|
| Steps | “Check if DB is healthy” | psql -h prod-db -c "SELECT 1" with expected output |
| Branching | “If it doesn’t work, investigate” | “If timeout >5s: skip to Step 7 (VPC)” |
| Testing | Never tested until incident | Runs against staging weekly |
| Freshness | Last updated 8 months ago | CI fails if referenced resources don’t exist |
| Automation | Human executes every step | Steps 1-4 automated, human decides at Step 5 |
| MTTR impact | Wildly variable | Minutes for known patterns |
A proper runbook step tells you what to do, what you should see, and what to do when you don’t see it:
Step 3: Verify database connectivity
Command: psql -h prod-db.internal -p 5432 -U readonly -c "SELECT 1"
Expected: Returns 1 row in < 100ms
If timeout (> 5s): Skip to Step 7 (Network/VPC investigation)
If connection refused: Skip to Step 5 (Instance health check)
If slow (100ms-5s): Continue to Step 4 (Connection pool check)
Time limit: 2 minutes. If unclear, escalate to database on-call.
Every executable step has five properties: the exact command, the expected output for a healthy state, branching logic for each failure mode, a time limit before escalation, and the specific person or team to escalate to. The pilot’s checklist. Read the gauge. Compare to the expected value. Branch. Go audit your runbooks and count the steps that have all five. Most teams are shocked at how few pass. (Fewer than they’d admit.)
The other trap: runbooks that assume the thing they’re diagnosing still works. Your runbook for “database connectivity failure” should not assume the database is reachable. Your runbook for “network partition” should not assume the VPN works. An emergency procedure for engine fire that starts with “engage the engine.” If the thing your runbook diagnoses is also a prerequisite for executing your runbook, the document fails exactly when you need it most.
Automation: Filling the Dead Window
The highest-value seconds in incident response are the first two minutes between alert and human acknowledgment. Most teams waste that window completely. The alarm sounds. The pilot is waking up. The instruments should already be telling the story. Fill it with automated diagnostics and the on-call engineer starts with context instead of starting from scratch.
- Alerting rules fire within 60 seconds of threshold breach
- Runbook platform has API-triggered execution capability
- Service metadata maps each alert to the correct runbook
- Automated steps have been tested against staging within the last 30 days
- Escalation contacts are current and validated monthly
Runbook automation platforms execute steps automatically when alerts fire, before a human even picks up the phone. For a team with a 5-minute acknowledgment SLA, those automated first steps separate a contained event from a cascading outage.
Diagnostic collection. Grab the last 15 minutes of logs, current CPU/memory/connection counts, recent deploys, upstream dependency health. Package as a structured incident brief. When the on-call opens the channel, context is already waiting. The cockpit instruments that read themselves and print a summary before the pilot touches the controls.
Known-safe remediation. Restart a pod with a high restart count. Drain a saturated connection pool. Clear a dead letter queue blocking a consumer. Scale up an autoscaling group at its ceiling. Deterministic responses to specific signals. The autopilot that handles turbulence while the pilot is still reaching for the controls.
Health verification. Poll health endpoints for 5 minutes after any automated action. System recovers? Notify the on-call with a summary. Doesn’t recover? Escalate with diagnostics attached. Automated remediation patterns pay for themselves on the first routine incident they handle without a human.
Don’t: Automate actions that could cause cascading failures. An automated restart during a schema migration compounds the incident. The autopilot that retracts the landing gear during taxi.
Do: Classify every automated action by blast radius. Restarts, drains, and diagnostic collection are safe. Data mutations, schema changes, and cross-service orchestration require human judgment. Automate what’s safe. Humans handle what’s dangerous.
The constraint is simple: automated steps must be safe without human confirmation. Data loss risk, cascade potential, or non-obvious side effects? Human in the loop. Automation handles diagnostics. Humans handle blast radius.
Coverage and Staleness: The Metrics That Predict Outcomes
| Maturity Level | Coverage | Freshness Cadence | Automation | MTTR Profile |
|---|---|---|---|---|
| Reactive | < 25% of alerts | No schedule | None | Wildly variable |
| Developing | 25-50% | Quarterly reviews | Diagnostic collection | Improving but inconsistent |
| Mature | 50-80% | 90-day validation cycles | Diagnostics + safe remediation | Minutes for known patterns |
| Advanced | > 80% | CI-integrated validation | Full triage automation | Seconds for automated cases |
Coverage is the share of alerting rules with linked runbooks. Most organizations sit below 50%, meaning the majority of pages hand the engineer nothing but an alert description and good wishes. An emergency with no checklist. Measure coverage by comparing your alert rule count against your runbook inventory. Start with the 20 alerts that fire most often. Those cover the bulk of page volume.
Freshness is harder to maintain because runbooks decay quietly. Services get renamed, endpoints change, instances get decommissioned, teams reorganize, and the escalation contact leaves the company. A checklist written for a 737 being used on a 787. Half the switches are in different places. The pilot discovers this during the emergency. A runbook referencing a hostname decommissioned six months ago sends the on-call engineer on a dead-end detour during the most time-pressured moment of their week.
The freshness discipline that works: tag every runbook with a “last validated” date. Set a 90-day threshold. Any runbook past the threshold gets flagged and assigned to the owning team’s next sprint. Run an automated check that verifies every hostname, endpoint, and command in the runbook is resolvable. When it fails, open a ticket automatically.
The more systematic approach: tie runbook validation to infrastructure changes. When a DevOps pipeline deploys a change to a service covered by a runbook, trigger a validation job that executes diagnostic commands against a test environment. If they fail, the deployment still proceeds (don’t block deploys on documentation) but a high-priority ticket opens with a 48-hour SLA.
The Post-Mortem Feedback Loop
Coverage and freshness keep runbooks useful. But the mechanism that makes them improve over time is simpler and harder: actually updating them after incidents. The accident investigation that rewrites the checklist.
Every post-mortem should answer four questions about runbook effectiveness:
Did a runbook exist for this failure mode? If not, create one within 48 hours while the failure is fresh. The investigation that produces a new checklist item.
Was the runbook followed? If the on-call engineer deviated, find out why. Usually the runbook assumed conditions that weren’t present. Design problem, not a people problem. The checklist said “read gauge 3A” but gauge 3A was removed in the last refit.
Did the runbook help? Track time-to-resolution for incidents where a runbook was used versus improvised. This gives you a concrete MTTR number to justify investment.
What was missing? Every incident teaches you something the runbook didn’t cover. A new failure branch. A diagnostic step that would have saved ten minutes. A faster escalation path. Capture these while the incident is fresh. (Fresh means 48 hours. After that, the details blur and the update never happens.)
The failure mode at almost every organization: post-mortem action items pile up and never get completed. Someone creates an item to update the stale runbook, and that item festers in the backlog for months. Break this cycle by enforcing a 48-hour SLA for runbook updates identified during incidents, and track completion rate as a team metric. Untracked post-mortem items are therapy, not engineering.
Combined with mature incident and change management practices, this feedback loop turns every incident into a concrete improvement to the next response.
Sample post-mortem runbook review template
## Runbook Review (attach to every post-mortem)
Incident ID: ___
Runbook used: ___ (or "none existed")
1. Runbook existed? [ ] Yes [ ] No → create within 48h
2. Runbook followed exactly? [ ] Yes [ ] Deviated → document why
3. Time saved vs improvising: ___ minutes (estimate)
4. Steps that were wrong or missing:
- Step ___: [description of gap]
- Step ___: [description of gap]
5. New failure branch discovered: [ ] Yes → add to runbook
6. Escalation path correct? [ ] Yes [ ] No → update contacts
7. Automation opportunity? [ ] Yes → file ticket with SLA
Owner for updates: ___
SLA: 48 hours from post-mortem close
Building the Runbook as a First-Class System
Version runbooks in Git alongside the services they describe. Test them against staging the same way you test code. Validate them when infrastructure changes. The checklist lives with the aircraft, not in a filing cabinet at headquarters. Mature site reliability practice treats runbooks as a system, not a wiki.
What the Industry Gets Wrong About Incident Runbooks
“Document the steps and you have a runbook.” A document that says “investigate the issue” is not a runbook. Executable runbooks include the exact command, the expected output, and the decision tree for each possible result. “Read gauge 3A. If below threshold, execute procedure 7.” The test: can a junior engineer who has never seen this failure follow it without improvising?
“Automate everything in the runbook.” Automate diagnostics and well-understood remediation. Keep the decision points human. An automated runbook that restarts a database during a schema migration causes more damage than the incident it was trying to fix. Autopilot that retracts the landing gear during taxi.
“If the runbook exists, it works.” Runbook existence and runbook correctness are completely different things. A runbook that references decommissioned hosts, outdated escalation contacts, or deprecated CLI flags is actively harmful. A checklist for an aircraft that’s been retrofitted twice. It sends the on-call engineer down a dead-end path during the worst possible moment. Existence without validation is false confidence.
Same page. Same engineer. An executable runbook that tests connection pool saturation, checks PgBouncer state, and surfaces the fix in the first 60 seconds turns a twenty-minute scramble into a non-event. The alarm sounds. The instruments already have the answer. The checklist points to the fix. The pilot’s job is to decide, not to investigate.