Skip to content
Menu

Disaster recovery readiness

Would your IT systems actually recover tomorrow?

Most organisations own a backup. Far fewer own evidence that a restore works, that someone knows the order, and that it finishes inside the time the business assumed.

Build a drill plan How to get in touch

Recovery timeline: fault, isolation, ordered restore, verification Illustration of drill method, not a measurement. A horizontal time axis runs left to right. Four layers sit on their own lanes: storage, database, application and edge, each depending on the one before it. A fault hits storage and climbs the dependency chain, failing database, application and edge in turn. The backup repository, kept apart from the fault, is the restore source. Layers are restored in the same order, storage first and edge last, and then become verified one by one after a verification pass. A marker shows the recovery time objective set before the drill; the gap between that marker and the moment service is verified is what the drill records. Illustrative sequence. Not a measurement of any live system. RTO, set before the drill Backup repositoryrestore source, kept apart Storagerestored 1st Databasedepends on storage Applicationdepends on database Edgedepends on application restore from backup Fault isolated to its dependents Gap the drill records Service verified Fault Impact Isolate Restore in order Verify Verified In service Failed Restoring Restored, unverified Verified Backup source Recovery timeline: fault, isolation, ordered restore, verification Illustration of drill method, not a measurement. A horizontal time axis runs left to right. Four layers sit on their own lanes: storage, database, application and edge, each depending on the one before it. A fault hits storage and climbs the dependency chain, failing database, application and edge in turn. The backup repository, kept apart from the fault, is the restore source. Layers are restored in the same order, storage first and edge last, and then become verified one by one after a verification pass. A marker shows the recovery time objective set before the drill; the gap between that marker and the moment service is verified is what the drill records. Illustrative sequence. Not a measurement of any live system. RTO, set before the drill Backup repositoryrestore source, kept apart Storagerestored 1st Databasedepends on storage Applicationdepends on database Edgedepends on application restore from backup Fault isolated to its dependents Gap the drill records Service verified Fault Impact Isolate Restore in order Verify Verified In service Failed Restoring Restored, unverified Verified Backup source Recovery timeline: fault, isolation, ordered restore, verification Illustration of drill method, not a measurement. A horizontal time axis runs left to right. Four layers sit on their own lanes: storage, database, application and edge, each depending on the one before it. A fault hits storage and climbs the dependency chain, failing database, application and edge in turn. The backup repository, kept apart from the fault, is the restore source. Layers are restored in the same order, storage first and edge last, and then become verified one by one after a verification pass. A marker shows the recovery time objective set before the drill; the gap between that marker and the moment service is verified is what the drill records. Illustrative. Not a measurement of a live system. RTO, set before the drill Backup repositoryrestore source, kept apart Storagerestored 1st Databasedepends on storage Applicationdepends on database Edgedepends on application restore from backup Gap the drill records Service verified Fault Restore Verify In service Failed Restoring Restored, unverified Verified
ITDrill publishes drill method and planning material. It does not test, monitor or attest to any reader’s systems.

The premise

A backup is a claim.
A restore you have watched finish is evidence.

  • OrderRecovery has a sequence. Each layer can only come back once the one beneath it has.
  • DurationWhether recovery fits the time the business assumed is only known by measuring a rehearsal.
  • PeopleA runbook nobody has followed under pressure is a hypothesis about people, not a procedure.

Claim

“The backup job reports success.”

Says that data was written. Says nothing about order, duration or whether anyone can drive the restore.

Evidence

“The restore was watched, timed and checked.”

Records the order followed, the time it took against the objective, and the check that proved the service worked.

Schematic contrast of two kinds of statement. Not data from any real system.

Recovery has an order because dependencies do.

A fault in storage takes down only what depends on it. Restoring in the order the dependencies run, from a backup kept apart from the fault, is what stops each layer coming back to nothing underneath it. A layer counts as recovered only once it has been checked.

Dependency layers: a storage fault, its blast radius, restore from backup, verification Illustration of drill method, not a measurement. Four dependency layers: storage, database, application and edge. An unaffected chain runs beside the chain that depends on one storage volume. A fault hits that volume. Only its dependents fail, in dependency order, and the blast radius is drawn around them. The backup repository, kept apart from the fault, is the restore source. The chain is restored from it in order: storage first, then database, application and edge. A verification pass then checks each layer. Illustrative. Not a measurement of any live system. 01 Storage 02 Database 03 Application 04 Edge restore source Unaffected chain keeps running Backup repository kept apart from the fault Volume A reporting data unaffected Volume B primary data restore 1st Reporting store reads volume A unaffected Primary database writes to volume B restore 2nd Batch services reads reporting store unaffected Application tier calls primary database restore 3rd Admin portal serves batch output unaffected Public edge routes to application restore 4th Blast radius: dependents of the fault only Verification pass Storage volume readable and consistent Database accepts and returns a test record Application answers a defined request Edge routes a request through to the application Service verified Fault Impact Isolate Restore in order Verify Dependency layers: a storage fault, its blast radius, restore from backup, verification Illustration of drill method, not a measurement. Four dependency layers: storage, database, application and edge. An unaffected chain runs beside the chain that depends on one storage volume. A fault hits that volume. Only its dependents fail, in dependency order, and the blast radius is drawn around them. The backup repository, kept apart from the fault, is the restore source. The chain is restored from it in order: storage first, then database, application and edge. A verification pass then checks each layer. Illustrative. Not a measurement of any live system. 01 Storage 02 Database 03 Application 04 Edge Backup repository restore source, kept apart from the fault Volume A unaffected Volume B restore 1st Reporting store unaffected Primary database restore 2nd Batch services unaffected Application tier restore 3rd Admin portal unaffected Public edge restore 4th Phase Fault injected Impact traced Isolated Restore in order Verify Verification pass each layer checked in turn Service verified
Illustration of drill method: a fault, its blast radius, ordered restore from backup and verification. Not a measurement of any live system.

What a drill is for

Four questions only a rehearsal can answer.

  1. 01

    In what order?

    Dependencies set the sequence. A layer can only be proven once the one beneath it is back.

  2. 02

    For how long?

    Whether recovery fits the objective is a measured duration from a timed run, not an estimate.

  3. 03

    Done by whom?

    A runbook is only a procedure once someone other than its author has followed it.

  4. 04

    Proven how?

    A check written before the restore starts decides what counts as recovered.

Recovery validation

Anatomy of a drill, from injected fault to recorded result.

A drill is a timed sequence with three people-shaped lanes and one service-shaped lane. Service is restored before it is verified, and only verified counts. The recovery time objective is a marker you set before it starts; the drill’s job is to record the distance between that marker and what actually happened.

Anatomy of a recovery drill: five phases, four lanes, an RTO marker Illustrative sequence, not measured data. Five phases run left to right: inject, contain, restore, verify, review. Lanes show what scenario control, the recovery team and an observer do, and the state of the service: in service, down from the fault, restoring, restored but unverified, and finally verified once the recovery checks pass. A vertical marker shows the recovery time objective chosen before the drill. The gap between that marker and the moment service is verified is what the drill exists to record. Positions are illustrative and a real drill may land on either side of the marker. Illustrative. Positions are not measurements. 01 Inject the agreed fault 02 Contain scope and isolate 03 Restore dependency order 04 Verify defined checks 05 Review record and assign Scenario control Recovery team Observer / scribe Service state Inject the faultas agreed in advance Hold the scenario, answer questionsdo not steer the team Detect and declarewho calls it, and how Scope the impactisolate the fault Restore in orderstorage up to edge Run recovery checksdefined first Walk the gapswhat slowed it Start the clockat the fault Log decisions, waits and hand-offs Actual vs objectiverecord, assign fixes In service Down Restoring Unverified Restored Verified RTO budget: from the fault to the objective set before the drill RTOGap the drill records Anatomy of a recovery drill: five phases and an RTO marker Illustrative sequence, not measured data. Five phases run top to bottom: inject, contain, restore, verify, review, with what scenario control, the recovery team and the observer do in each. A bar on the left shows service state: in service, down, restoring, restored but unverified, then verified once the recovery checks pass. A marker shows the recovery time objective chosen before the drill. The gap between that marker and verification is what the drill records, and a real drill may land on either side of it. Illustrative. Positions are not measurements. 01 Inject Controlinject the agreed fault Teamdetect and declare Observerstart the clock 02 Contain Controlhold the scenario Teamscope impact, isolate fault Observerlog decisions and waits 03 Restore Controlanswer, do not steer Teamrestore in dependency order Observerlog each wait 04 Verify Teamrun the checks defined first Observercompare with the objective 05 Review Teamwalk through the gaps Observerrecord actual vs objective Service RTO, set before the drill In service Down Restoring Unverified Verified
Illustrative sequence of one drill. Positions are not measurements, and a real drill may land on either side of the RTO marker.

Read the full drill design guide RTO and RPO explained

How a drill runs

A drill is not a backup check. It is a rehearsal with an outcome you record.

Each stage below depends on the one before it. Skipping straight to the restore is the most common reason a drill produces no usable evidence.

Read the full drill design guide

  1. 01

    Define what “recovered” means

    Name the service, not the server. Write the condition that proves it is back — a transaction completing, a report generating — before anyone touches a restore.

  2. 02

    Choose the fault, and who knows about it

    An announced drill tests the runbook. An unannounced one tests the organisation. They answer different questions, so decide which you are asking.

  3. 03

    Restore in dependency order

    Storage before database, database before application. Teams that restore the customer-facing tier first routinely discover it has nothing underneath it to talk to.

  4. 04

    Record the elapsed time against the objective

    A drill that finishes without a measured duration has not tested the RTO — it has only tested that the data exists. Write down where the time actually went.

Next step

Plan the drill before you need the recovery.

The drill planner turns a service, a fault and an objective into a written plan you can run. It produces a plan, not a result: nothing it outputs is a certified recovery plan or an audited RTO or RPO.

Build a drill plan How to get in touch