Hurricane Sandy, the second costliest hurricane in United States history, is an object lesson in how the best-laid contingency plans can be shredded in no time when an extreme weather event hits. The storm has spurred a new way of thinking about data center resiliency
Sandy was the "acid test" for data centers wanting know whether they had a comprehensive disaster recovery plan, says Mark Evanko, principal engineer at BRUNS-PAK. Evanko cites instances of data centers going down that also took down operations in Chicago and Dallas. Hospital complexes in the New York metro area had to evacuate because power and communications went down.
During the storm, water rose 13 feet above sea level, which caused New York City to expand its flood plain area and redraw its flood maps. A lot of basements in multitenant data centers were flooded, and that is where much of the critical infrastructure was located.
New York City code requires fuel to be stored at the bottom of buildings, which is why fuel pumps, tanks, and generators, were in the basement. "Some facilities have generators on the roof," says Michael Levy, analyst, data centers, for 451 Research, "but the problem was that fuel pumps and fuel tanks got knocked over, and they couldn't bring fuel from the basement to the roof." UPS and transfer systems at higher levels didn't do much good when basements had floor-to-ceiling flooding.
Robert Ioanna, vice president, Syska Hennessy Group, describes similar problems when referring to smaller data centers in high-rise buildings downtown, where pumps wouldn't function and power feeds weren't watertight. "When water filled up basements, the oil tanks lifted off their carriages and piping connections to fuel oil were broken," he says. Trucks came in with package pumping systems that were physically connected to the fuel oil risers in buildings to run the generators.
Some generators were down several days, says John Bredehorst, managing director for WSP/Flack + Kurtz, but portable generators were brought in after the water was cleared. Getting portable generators, which were in great demand, was more of a challenge than obtaining fuel, which was available.
One success story involved a company that took prior action to hold the water back, Bredehorst said. People welded doors on the lower level so that water couldn't get in, and built a wall in front of the fuel tank.
Generally, data center backup power is tested for short periods of time and is not run for 10 days or more, as it was in some data centers during Hurricane Sandy. The backup systems were tried "to their breaking point," says Ioanna. Fuel-oil filters are one of many maintenance items on generators. "If the fuel oil is dirty and it passes through the filter, you have issues," he says.
For some data centers, salt water rendered equipment useless and in need of replacement, so generators that were not equipped to run for extended periods of time had to be taxed to the limit.
In some cases when data centers went down, operations were picked up by other data centers run by the same company. For example, some financial transactions went to "dark trading sites" that do not operate at 100 percent capacity, Ioanna explains, a less-than-optimal choice.
Hurricane Sandy Prompts New Way Of Thinking About Data Center Resiliency
New York City Housing Authority Data Center's Backup Power Strategy Helped Weather Sandy
Hurricane Sandy Shows Value Of Cloud Services, Co-location And Redundant Facilities
Hurricane Sandy's Impact Has Data Centers Outside New York Re-evaluating Threats
