Even the most experienced and thorough facility managers can get caught off-guard from time to time. In a department that includes critical facilities such as data centers, adjusting to those surprise circumstances successfully can result in saving hundreds of thousands of dollars for facilities, not to mention preventing the loss of important data.
In some cases, those surprises are related to a data center infrastructure technology like the uninterruptible power supply.
"A limited risk most data center facility managers face is the potential for a UPS battery thermal runaway event," says David Boston, director of facility operations solutions for TiePoint-bkm Engineering. "Although uncommon, we hear of several events periodically — mainly through consortiums of critical facility managers who report on risks encountered to the rest of their group. The situation most frequently occurs in valve regulated acid batteries, when the rate of internal heat generation becomes greater than the rate at which heat can be expelled. Chemicals released during thermal runaway can be toxic. Thermal runaway can also create electrical fires and explosions."
Consistent preventive maintenance, including inspections, Boston says, can help prevent this issue.
"Preventive maintenance can identify most problems with batteries before they fail," he says. "Monitoring charging voltage and current will help confirm batteries will not overcharge. Maintaining proper air temperature in the battery room is also important. Thermal runaway may still occur when all of these practices are followed. It is important to plan for the potential event and have procedures in place to permit safe and rapid response when an over-temperature alarm or an inspection reveals the condition."
Surprises in the data center environment can be costly. Consider what can happen when an inert gas fire suppression system discharges. “The discharge can produce a loud, shrill noise and vibration that has been documented to damage spinning drives in servers and storage equipment,” says Michael Fluegeman, director engineering at PlanNet. “These big tanks are filled with compressed gas, and it's not like a little bit comes out. It's a full-on dump. It's like a tornado inside a data center.” It’s not a widespread problem but it’s one that facility managers should be aware of.
It’s one thing if the system discharges as a result of a fire. But false alarms can also trigger discharges, Fluegeman says. And there are other circumstances where the gas can be discharged even if there isn’t a fire. Fluegeman cites a company that installed a gas system in its live data center. “The fire marshal said he needed to see it work, so they brought the vendor in and he discharged it,” Fluegeman reports. “It crashed all the hard drives. They had a lot of damage."
Changes in compliance responsibilities may also catch facility managers off guard. Jeff Gilmer of Excipio Consulting says that data center facility managers now have to be aware of compliance and regulatory issue responsibilities that have started to be thrust upon facility managers.
"The facilities people have always let the business people take care of that," Gilmer says. "Now, it's coming to the facilities people.” The fines are significant. He cites one company that for the past five years have not been able to get their data centers or servers to meet the compliance requirements. “They have paid over $25 million in fines because they can't get this system to meet the compliance requirements that they have," Gilmer says.
Gilmer also notes that, as many facilities move to a cloud-based or colocation site for their prime data centers, facility managers may be surprised to know that the old data centers aren't necessarily obsolete.
"That production data center is still a great asset if people want to look at it from a disaster recovery standpoint," he said. "People shouldn't be thinking 'we're going to cloud, we're going to colo, let's throw out the production data center and just waste the money.' Look at it as an opportunity to be the disaster recovery center. There are ways to reuse that old facility rather than throwing out all the assets and writing off the costs. Disaster recovery is a great way to do it."
Dave Lubach has covered a wide range of facility technology topics. He was formerly the associate editor for Facility Maintenance Decisions magazine.
David Boston, Michael Fluegeman, and Jeff Gilmer are speaking at the 2018 Critical Facilities Summit.
