Once facility staff have calculated the acceptable downtime and the impact of any downtime, as part of its analysis of business-critical operations, fact-based decisions can be made about a critical operation. Using all the information gathered, a gap analysis can be completed to compare the designed operating time of redundant systems against the acceptable downtime. If the acceptable downtime is greater than the designed operating time, this indicates an area of concern.
In many applications, several systems are engaged in sequence to provide a more robust and longer designed operating time. For example, if the utility power is interrupted, an uninterruptible power supply will carry the critical load until the generator takes over and produces the electricity needed to operate the systems. Each system has a limited operating time, and it is important to identify the individual system’s design limitations in order to understand the implications of a single system failure.
In some instances additional or enhanced redundant systems will be required, while in other instances the identification of alternate processes or paths will be the best method. A financial analysis model can document the cost of downtime versus the cost of the systems. This considers the cost to install or upgrade and historic frequency versus the impact of the interruption. For example, a business output may have an eight-hour acceptable downtime with an impact of $100,000. If power is lost, the business will not be able to operate. Currently there are no redundant systems. Installing a generator will cost $250,000. If the building historically has never lost power for more than one hour, then it would appear that the return on investment is not there. Historically, however, if power is lost two to three times during the year due to storms, and the outages last 12 to 24 hours each, then there is a very strong financial case for installing the generators.
Sometimes services are not duplicated using redundant systems, but by identifying alternate paths, processes, or procedures. Shifting the operation temporarily from one location to another changes the path. Going from an automated system to a manual system provides for an alternate process, and enacting temporary procedures — such as having a person manually control the chiller plant if the automated system is disrupted — can provide the required redundancy. The organization also has to decide whether to prepare for the most likely event rather than a catastrophic failure. This could be having a UPS that will carry the load for 15 minutes in the event of a power failure, if the historic power failures are only two to five minutes in duration. These options will need to be discussed, and a recommendation agreed upon, by the senior management of each organization.
The analysis of business-critical operations for fully understanding the options available is essential in risk mitigation. Effectively identifying, comprehending, and documenting the full impact of these operations will enable the appropriate planning required to minimize risk and ensure the continuing business operation in the worst scenarios. Understanding the critical operation, analyzing the services and systems needed to support the operation, and understanding the system design limitations provide the information needed to analyze the potential exposure. Formally identifying the owners of the systems and creating redundancy plans using either duplicate systems or alternate processes is the foundation of a risk mitigation program.
Scott Offermann, managing director, critical operations, Cushman & Wakefield, has more than 25 years of facilities management and critical operations experience in data centers, labs, healthcare, distribution, high tech, recreation, office, and call center environments. He can be reached at scott.offermann@cushwake.com.
Understanding Whether a ‘Critical Operation’ is Really Critical
Identifying Levels of Support, Measuring Productivity Loss Help Determine ‘Acceptable Downtime’
In Study of Critical Operations, Gap Analysis Examines Operating Time of Redundant Systems
