Q. If you're facing a situation where you can't prevent damage or disruption, what options do you have for being able to share the load with short notice?
A. It is critical to not have all your data in one location or even in two locations that are susceptible to the same disaster event. It is important to have a predetermined agreement with another data center location or hosting facility where you can transfer your operations. If your facility employs a synchronous configuration, you can move your business critical operations to the other site or to your disaster recovery facility (assuming it is not in the same region as the event).
Q. If you have an unexpected failure due to worse than anticipated weather, how can you quickly bring service back up? What can you do both onsite and off-site to expedite recovery?
A. There is really not a lot you can do; that’s why preparedness is so important. Having a good disaster preparedness and recovery plan is critical. You should already have pre-determined arrangements with other facilities to provide for payroll, communications, customer services, shipping and receiving, information systems, emergency power, and recovery support. A good disaster recovery plan will include “what if” scenarios such as this and a plan of action to mitigate any business impacts.
Q. If you have an event like Hurricane Sandy and find out that your plans aren't sufficient, where do you begin when it comes to evaluating how to improve them?
A. If it is determined that your disaster preparedness and recovery plans are inadequate, then immediately start over (what else have you missed?).
When formulating a plan for disaster preparedness, consider how different disasters will impact your business and categorize the likely threats to the organization. Conduct a Business Impact Assessment (BIA) and risk assessment to determine if the priority of business functions and establish classes of recovery in terms of Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO). Develop an accurate inventory of IT assets to understand the resources and business processes that must be protected.
Determine service-level agreements and define the uptime and recovery parameters for the organization’s software and hardware assets. Once this is established and agreed upon, determine what methods are available to remediate the deficiencies.
The cost of the disaster response should be weighed against the business tolerance for system downtime. The higher the system availability needs, the more it will cost to develop a suitable response.
Some organizations have a data center disaster recovery strategy that utilizes a “systems redundancy” solution. Multiple systems (mainframes and servers) run in production and backup data facilities. If a system in the production environment runs into problems, it immediately fails over to the local backup as a first line of defense. This solution is effective, but also very expensive.
Consider bringing in an outside consultant that specializes in data center disaster recovery to help you assess your plans and make recommendations to mitigate any risks.
Other Resources:
Making Informed Decisions About Uninterruptible Power:
https://www.facilitiesnet.com/emergencypreparedness/article/Making-Informed-Decisions-About-Uninterruptible-Power--12879
Emergency Power: The ABCs of UPS
https://www.facilitiesnet.com/emergencypreparedness/article/Emergency-Power-The-ABCs-of-UPS--8596
Portable Cooling: Strategies For Mission-Critical Facility Operations
https://www.facilitiesnet.com/datacenters/article/Portable-Cooling-Strategies-for-MissionCritical-Facility-Operations--13580
Q&A: Data Center Expert Chris Wade on Disaster Preparedness, Planning
What To Do When You Can't Prevent Storm Damage To Your Data Center
