MHA planned and conducted an unannounced enterprise wide live mock disaster exercise for its leading financial risk management firm client to functionally validate the ability of its Crisis Management Team, Recovery Teams and Associates to work completely independent of its headquarters facility for a day and maintain functional and compliant functions to its clients across the United States. The firm processes work for all of the major financial institutions across the country.
The client started their BCP program formally in 2007 by hiring MHA Consulting to fully manage their BCP Office. Over the five years, the client and MHA have built a highly robust BCP and DR program that has no single points of failure. It is a highly regulated organization, requiring recovery of its critical business processes, systems and applications in hours and with no data loss. A decision was made by senior management to require an enterprise wide recovery exercise simulating Loss of Building Access and running the entire company using its virtual work strategy for a day. Here is an outline of key events on that day:
- 4:00 am: notification is made to the Crisis Management team via the Emergency Notification System (ENS) to attend a 7 am conference call.
- 4:03 am: initial notification from the Emergency Notification System (ENS) goes out to all employees locally and nationally that a live mock disaster exercise is taking place and all production work will take place from home using the virtual work at home strategy.
- 6:00 am: The Virtual PBX is activated and all incoming calls to employees and business units are forwarded to pre-defined numbers in the database.
- 6:30 am: Employees who did not receive the message are advised of the exercise, turned away from the office and asked to contact their managers for information.
- 7:00 am: Crisis Management Team holds its initial conference call and assesses the simulated event.
- 7:00 am: Business units follow Loss of Building Access scenario in their business recovery plans. Business units begin contacting key vendors and others who were scheduled to come to work and make alternative arrangements.
- 8:00 am: Business units run production operations connecting to systems in the east coast data center for the day. The west coast backup data center remained fully operational and ready if needed.
Additionally, the Crisis Management Team held regular status calls to assess the status of the recovery exercise and impact to its customers. Corporate Communications advised clients of the exercise and no impact to production services and functions. The Call Center and Help Desk continued to take calls using the virtual work strategy. The Board of Directors were notified of the exercise and updated as needed.
Key opportunities for improvement included:
- Ensure laptops, chargers, power cords, etc., are taken home each night.
- All employees need to test use of the Virtual PBX capability and VPN access at a minimum of once per quarter.
- Always keep personal contact (e.g., home, cell, other) current within the HR Service Center, Emergency Notification System and Virtual PBX.
The reason for the success of the exercise is simple. The client has integrated BCP into its culture; it is not a “nice to have”, its a part of everyday business and commitment to employees, customers and stakeholders.