Under the new policy, the facilities department is the sole owner of all network cable, fiber, and power cord installation and removal in the data centers. No other individuals physically touch a cord. The number of people with access to the computer rooms was methodically reduced.
Other elements of the new process:
• PDUs and RPPs are never loaded more than 40 percent and loads are always balanced between PDUs or RPPs, within a pair.
• Installation and removal of devices is only conducted during a nightly window between midnight and 6:00 a.m. All cords are traced before device removal. All cords connected to a PDU are traced prior to each planned PM on the PDU (one power path will be shut down). The B-side cord in each dual-cord installation is identified with red tape affixed to the cord above the plug at each end. Single corded devices are identified with a piece of blue tape affixed above the plug on the cord at each end. A comprehensive PDU circuit survey procedure and form is employed during each device installation and removal activity. (See “Step-by-Step: PDU Circuit Survey” on this page.)
• Third party vendors, which sometimes pre-configure a rack when many devices are added at once, must adhere to a detailed process developed by the data center owner. Facilities technicians inspect the pre-configured rack prior to applying power.
• A defined process is used for tracing circuits, requiring each cord to be followed by hand within a designated cabinet from the ATS or power strip to the device, to confirm it is connected as recorded. A defined process is also followed for shutting down a PDU during a PM or repair activity. Facilities technicians verify the second PDU in the pair is loaded less than 40 percent and infrared scan the second PDU in the pair to identify any loose connections or hot spots. Then the feeder breakers on the PDU to be maintained are opened. The technician infrared scans the second PDU in the pair for 30 minutes to observe any excessive heat rise. The facilities technician backs out and re-powers the other PDU if any breaker in the second PDU begins to overheat. A defined process is employed for powering a PDU following a PM or repair activity: Panel board mains are left off. The feeder to the PDU is energized. The facilities technician verifies closed transition, panel board mains on, and loads accepted.
Benefits Realized
Since these changes were executed, the organization has eliminated surprise power interruptions related to device installation and removal. The circuit tracing process has proven its value by exposing a few devices over the years that were incorrectly connected. However, the other components of the new process have ensured that these discoveries are rare. This record of continuous power availability within the computer rooms at this organization is exceptional, particularly in light of the high frequency of device installations and removals.
This project illustrates the value of clearly articulating a problem and proposed solution; gaining support from multiple departments who share a common objective; and consistently deploying a substantial change in process. As one of the owner’s facilities senior technicians likes to point out: “Poor power cord management will ensure disappointment with the best designed critical power systems. Management of the process is the key to success, as well as the most economical means of achieving the desired reliability record.”
David Boston is director of facility operations solutions for TiePoint-bkm Engineering, where he provides assessments, staff plans, and development of comprehensive procedures and training programs for critical facility clients. He can be reached at DBoston@TiePoint-bkm.com. Previously, at the Uptime Institute, he assisted data center management teams for 14 years in two roles: consortium director and consultant. Boston began his critical facilities career as facilities manager for GTE Data Services from 1985 through 1995.
Alan Hart is currently a senior data center facilities technician providing operational guidance, technical support and training for the technicians for a major financial services company. Other duties include project oversight and operational expense budgeting for the team. During his tenure there he has become the subject matter expert for UPS, critical power distribution, batteries, physical security, and fire detection and suppression systems. He is an original member of the data center facilities team dedicated to providing 24x7x365 primary support for all critical infrastructures within the two data centers on the headquarters campus.
Email comments and questions to edward.sullivan@tradepress.com.
In Data Centers, Human Error Is Most Common Cause of Downtime
Facilties Department Solely Responsible For Network Cable, Fiber, and Power Cord Installation
SIDEBAR: Survey Says Facility Staff in Charge of Data Centers Earn Higher Salaries
