One key consideration to reliable operations in critical facilities is the challenge of staffing issues. Although the data center industry has been successful at "hardening" facilities and physical infrastructure, it has not done as well with the associated operating staff and facility management aspects.
It is widely recognized that the vast majority of critical facility problems can now be attributed to human error (some sources claim as high as 70 to 80 percent of problems). There has been a direct correlation between the increase in infrastructure complexity and the increase in human error by the operating staff.
The problem here is not one of availability. Most critical facilities have staff on-site continuously (100 percent availability). The problem is in staff reliability (and in some cases validity). Unlike computers, people get tired, distracted, sick, confused, etc., which can all lead to unreliable performance.
The answer is to have processes that produce reliable results. Detailed, step-by-step procedures are a good example, but unless they are followed correctly each time there is no guarantee of a reliable outcome. On the other hand, if operating staff is required to initial a checklist as each step is completed, and to have the action witnessed by a separate participant, the reliability of the process improves greatly. If the procedure also describes the expected outcome or result associated with each step, such as the expected pressure and flow when starting a pump, or the expected indicating lights and annunciations when closing a breaker, then the validity of the process is ensured. Operating staff who blindly follow a procedure without equal attention to the results will inevitably produce unintended outcomes.
Performance-based training is also a process. Training a new computer or controller is simple and quick. Download the programming, connect to the network, and the new computer is 100 percent as capable as the failed computer or controller that it replaced. Training new staff isn't quite as easy. People are individuals. Each of us is a one of a kind.
