Beginner’s Guide to Recovery Exercises

What business continuity or disaster recovery exercises have your performed? Do you know the difference and have distinct goals for them? Do you even do exercises? We receive many questions on this topic, to the point where we thought it might be helpful to devote today’s post to a “Beginner’s Guide to Recovery Exercises.”

In this overview, we’re going to provide some introductory information on this essential topic. Specifically, we’re going to answer the following questions:

• Why perform recovery exercises?
• What are the benefits of performing recovery exercises?
• What are the three main types of exercises?
• How do you develop a testing program?
• What are the two areas where we perform recovery exercises?
• Do you perform exercises for business continuity and IT recovery separately?
• What exercises should your organization be performing?
• Do companies graduate beyond the need to do tabletop exercises?
• How frequently should the different types of exercise be performed?

Although we’ve titled this a “beginner’s” guide, we believe much of the information will also be helpful to those with more experience in conducting recovery exercises. We hope those folks will read on – we believe they will find the post informative. As we have said in previous posts, frequent reminders are important to ensure we stay on top of important aspects of BCM.

Now we’ll take the above questions one by one, starting from the top.

Why perform recovery exercises?

If you are not performing exercises to make sure you can recover your business in the event of a disruption or disaster, then you have no way of knowing that the recovery plans and strategies you have put in place will actually work if called upon. You only hope they will work. If this is sufficient for you and your organization, you have no need to perform recovery exercises. However, if you would rather base the future of your organization on something stronger than hope, you should be performing such exercises. The good news is, it’s never too late to begin.

Remember, hope is not a strategy.

What are the main benefits of performing recovery exercises?

There are several benefits of doing recovery exercises. They are:

1. Validating your recovery strategy.
2. Validating your recovery processes.
3. Identifying gaps in both your recovery processes and strategy allowing you to correct them before a real event occurs.
4. Training your staff so people know what to do. Actions performed during a recovery are different than day to day activities. We need to practice.

The Various Types of Recovery Exercises

Here’s a quick six-part taxonomy for business continuity and disaster recovery exercises. Terms vary, but we hope this clearly defines the scale of each exercise type.

Information technology and disaster recovery (IT/DR) exercises are technology-based. They focus on recovering processing functions, applications, systems, and data centers. These exercises look to see whether the organization can restore its technology and get it running again. They are very important but are not the be all end all.

Business continuity (BC) exercises focus on the actions taken in recovering your business processes, such as manufacturing, research, finance, and accounts payable (other than IT recovery). This can involve the ability to relocate people or processes to a new facility, if necessary.

The above exercises are conducted using the following methods (see the FFIEC Business Continuity Planning Booklet for additional information):

Tabletop exercises and structured walk-through tests are considered to be a preliminary step in the overall testing process and may be used as an effective training tool. The primary objective of this type of exercise is to ensure that critical personnel from all areas are familiar with the various plans (Business Continuity, Crisis Communication and Management, IT Recovery, Emergency Management, etc.) and that the plans accurately reflect the institution’s ability to recover from a disaster.

Walk-through drill and simulation tests (sometimes simply referred to as a “mock disaster exercise”) are meetings where the participants walk through how the organization would respond to an emergency scenario if it were to happen in reality. These DR exercises look at how management would assess the impacts of events, communicate among the parties, and determine whether to implement recovery procedures such as relocating to an alternate site. The vast majority of exercises performed fall into this category.

Functional exercises, functional drills, and parallel tests are the first type of test that involves the actual mobilization of personnel to other sites in an attempt to establish communications and perform actual recovery processing as set forth in the BCP and IT Recovery Plan.

Full-interruption and full-scale tests are the most comprehensive type of test. In a full-scale test, a real-life emergency is simulated as closely as possible, and the participants carry out a plan in the real world in real time, performing an actual failover from production locations or processing. This is much rarer and should only be performed once your tabletop and simulated recovery exercises have demonstrated full capability with very few or only minor issues.

Tip for Starting Out with Recovery Exercises

Start small and ramp up. Run tests based on the maturity of your program. Each of the tests is a kind of training before you move on to the next level. At each level, the stakes are higher, and the activity more closely replicates the situation of an actual disaster. Each level provides feedback and an opportunity to improve your procedures.

Start with a limited scope. You could do individual tabletop exercises with each department first. Then bring in multiple departments where dependencies exist. Make sure your processes and strategy seem sound before you go on to a simulated recovery.

Even within the different types of exercises, you’ll need to progress over time. With simulated recovery, for example, you’ll want to start with a few applications or business units then ramp up as you become more proficient.

Very few organizations actually perform a production recovery exercise. It’s too risky and most organizations have not done the necessary planning and preparation. This planning and preparation which comes from performing the other two exercises multiple times over the course of many years. Even then, be sure to consider the risk to impacting production if something goes wrong.

[vc_row][vc_column][vc_message icon_fontawesome=”fa fa-book”] Need more help making the most of your DR exercises? Discover More on Our Guide. [/vc_message][/vc_column][/vc_row]

What are the two areas for which recovery exercises are performed?

The areas for where we perform recovery exercises:

1. Business Continuity. These are the procedures for recovering the parts of the business other than information technology—so, departments such as finance and human resources and the rest: everything not IT. These folks’ focus during these exercises will be drilling on what they would do during an event to keep things going. Will they leave the facility and take equipment with them? Work at home? Make phone calls to customers or other employees? There may be IT or technology components, but these are things like phones, workstations, printers, etc. – anything other than applications or processing that occurs in the data center.
2. Disaster Recovery. This is the data center IT side. This is all about recovering the applications and technology that support the business

Are exercises for business continuity and IT recovery always performed separately?

In the beginning, probably. Mature organizations will exercise both of these areas together at some point. You’ll declare a mock disaster and while the IT team performs the recovery of your apps and technology, the business team will be performing their functions. These integrated exercises take a lot more planning than just working on one side or the other. The teams involved will talk about dependencies and you must clearly define the scope of the exercise.

It’s often the case that an organization’s testing is more mature in one area than another. Typically, the IT side (DR) is ahead of the business side in terms of preparedness, processes, and documentation. It might happen that in your joint exercise the DR team, for example, performs a simulated recovery while the BC side does only a tabletop exercise.

An integrated exercise is something you work your way toward over time. Once you’ve done tabletops and increased the scope to include multiple apps and environments (this could require five or more smaller exercises), then you could consider an integrated test where you bring DR and BC together and run a combined exercise that leverages both plans. For a company just starting an exercise program, such a project might be two to five years down the road.

What exercises should my organization be performing?

This mostly depends on where you are in your exercise program. You have to walk before you can run. If you’ve yet to do tabletops, that’s the place to start. When you have validated your strategies through tabletop exercises, you’ll be ready to move on to simulated recovery exercises. Then, maybe to production recovery exercises. When you have substantial experience in testing both sides of your business (BC and DR), you will be ready to think about conducting an integrated exercise as described above.

Do companies graduate beyond the need to do tabletop exercises?

Nope. Even companies with mature exercise programs can reap dividends from performing tabletop exercises. They are an underused resource in our exercise methodology. They’re easy to schedule and perform, take very little time, and bring ongoing benefits. Even after we learn to run, we still find it advantageous to walk a good deal of the time. These exercises are a good way to keep people thinking about DR/BC as well as to verify changes to plans, strategies, and processes when significant changes occur in the IT or business functions.

How frequently should the different types of exercise be performed?

Tabletop exercises performed within a single department you can do as time and resources allow. If you were doing a separate exercise for each department, you might do them quarterly.

Most organizations will do one to two major exercises a year. As you increase the scope of your exercises, they become more difficult to coordinate and execute. Depending on your strategy, you might do a smaller scale exercise once a quarter with more major scoped exercises annually. Exercises demonstrating your overall recovery strategy for both BC and DR should be performed at least annually. Depending on your recovery strategies, you may be able to perform smaller exercises much more frequently.

We hope this “Beginner’s Guide to Recovery Exercises” was helpful, whether you are new to BCM or more experienced.

In the meantime, please feel free to email us with your questions.