Caynetic Blog

The Outage Playbook for Customer-Facing Operations

A one-page outage playbook and synthetic POS recovery drill with safe reconciliation.

Back to Blog

Practical software guides

When a business app fails, staff need a short answer to four questions: what continues, what pauses, how customers are informed and how work re-enters the system. Store that answer somewhere that survives the same failure.

One-page outage playbook specimen
DecisionWrite your answer
Critical taskWhich customer service must continue?
Stop conditionWhen is work unsafe or unauthorised?
FallbackIndependent paper or read-only roster; authorised minimal data
CommunicationWho issues the message and when is the next update?
Recovery objectiveHow much lost work and downtime can this task tolerate?
Re-entryWho matches temporary references and resolves conflicts?

A fictional shop drill

Use a test till and five synthetic orders. Disconnect its network. Staff can use the approved price list and record orders with unique temporary references. A payment whose outcome is unknown stays unknown until checked with the provider; do not charge again merely because the screen timed out. Never write card details into a paper ledger or custom logs.

Restore connectivity. Match each temporary order once, reconcile the provider’s transactions and flag any quantity or price conflict for review. A duplicated order must not duplicate fulfilment. Clear temporary copies according to the approved retention procedure. Record observed restoration time and unresolved items; this walkthrough is a proposed drill, not a reported result.

Check independent failure domains

A web fallback hosted beside the failed app may fail too. A local roster can become stale and must be protected. A backup that no one has restored is still untested. Run separate tests for lost internet, an unavailable provider, a lost device and a corrupt restore.

The Ready.gov business preparedness guidance offers general planning context. For security incidents, use the incident recovery guide; for ticketing, use the admission-specific runbook. Equipment isolation and safety decisions require the responsible specialist, not instructions improvised from a blog.

Explore the current journal · Earlier topics