Writing a real DR runbook (before 2 AM)

Disaster recovery isn’t a backup — it’s the documented, rehearsed procedure for turning a backup into a running service under stress. The test isn’t whether you have a plan; it’s whether someone who didn’t write it can execute it at 2 AM.

What a real runbook contains

• Triggers — the specific conditions that declare a disaster (primary DC unreachable N minutes, data corruption, etc.), so nobody debates whether to act.
• Roles — who declares, who executes, who communicates. Decided in advance, not during.
• Step-by-step recovery — exact commands and order: promote the replica, repoint the origin, activate the standby, verify playback. No prose, no guessing.
• RTO / RPO — the target recovery time and acceptable data-loss window, so the plan is measured against a goal.
• Verification — how you confirm service is actually restored (not just ‘servers up’ but ‘a real device plays’).
• Communication — templated messages to subscribers and partners, because silence loses trust.

RTO and RPO drive the design

The part everyone skips: testing

An untested runbook is fiction. Rehearse it — a scheduled failover drill, ideally a real game-day — because the first time you discover the replica wasn’t actually replicating should not be during a real outage. Every drill finds something; that’s the point.

Keep it alive

Infrastructure changes; runbooks rot. Review after every architecture change and every drill. A runbook that describes last year’s system is worse than none, because it’s trusted and wrong.

Lessons

• A DR plan is a rehearsed procedure, not a backup or a paragraph.
• Include triggers, roles, exact steps, RTO/RPO, verification and comms.
• RTO/RPO decide your architecture — set them first.
• Untested runbooks are fiction; drill them and keep them current.