Recovering consistently and testing DR

Restarting from a crash-consistent copy

With good replication, the recovery site has disks that look as if the primary lost power at a single instant. z/OS is started (IPLed) from those disks, then the subsystems restart. DB2, IMS, CICS and MQ each use their logs to back out units of work that had not committed and to complete ones that had. This is the same restart they perform after any unplanned outage, which is one reason the approach works.

The batch problem

Online transactions are usually the easy part because each one is a unit of work that either committed or did not. Batch is harder. A disaster at 02:30 may hit with dozens of jobs part-way through, some updating VSAM files or sequential datasets that have no log to roll back. After recovery you must decide, job by job, what state the data is in.

• Which jobs had finished? The scheduler database must itself be recovered consistently with the data, or its view will not match reality.
• Which were running? Their outputs may be partly written. Check for checkpoint and restart records, and use the same rules as for any rerun or restart.
• Which inputs arrived? Files transmitted from partners just before the disaster may need to be resent.
• What about generation datasets? GDG relative numbers depend on the catalog; confirm the right generations are referenced.

Roles and runbooks

A disaster is the worst time to work out who decides what. DR plans define roles in advance, with deputies:

Each role works from a DR runbook that lists steps, checks and contacts. Runbooks must be available when the primary site is gone, so they are kept somewhere that does not depend on it, often printed copies as well as an offsite system.

Testing DR

An untested DR plan is a hope. Tests range from cheap to realistic:

Many sites test at least annually, and regulators in financial services often expect evidence of regular, realistic tests. A useful test measures actual recovery time and data loss against the RTO and RPO, and every gap becomes a tracked action, just as in a post-incident review.

Common DR failures

• New volumes, applications or datasets added to production but not to replication or consistency groups.
• Expired passwords, certificates or licence keys that only matter at the recovery site.
• Hardware differences at the recovery site, such as fewer processors or different network addresses, that stop work starting or meeting WLM goals.
• Runbooks referring to people who have left or systems that no longer exist.
• Recovery of the mainframe tested alone while dependent distributed services were never included.
• No tested plan for coming back to the primary site afterwards.