A Building Operating Management survey asked facility managers to share success stories and lessons learned from data centers that survived Hurricane Sandy. Their brief answers, listed below, provide tips that may help other facility managers better prepare for disasters, and offer a glimpse of what it took to survive the storm. The first group of responses has to do with power issues; the second deals largely with staffing and management.
But first, one quick story to set the stage:
"A Verizon message warning of high winds and advisory to stay indoors popped in on the cellphone while I was trying to open the roof door and check the diesel fuel level. The wind died down for a couple seconds and I managed to beat the great vacuum, crack it open only to have it fly back at me in a split second. If I did not withdraw my arms from the doorway... Well, you get the picture."
"Although our data center did not go down, we had to bring in additional generators to get the plant operational to bring employees into work and to provide a safe haven for them."
"Several of the filters on the generators became clogged and we almost ran the tanks dry. It was a challenge getting fuel. The sludge from the bottom of the tanks ended up in the filters."
"Need to make sure fuel pumps feeding emergency generators are safe from flooding."
"Procuring portable generators and fuel tanks versus rental has shortened our hurricane ready time."
"Diesel oil for the generators was an issue. Were able to get it resolved."
"Both our primary and secondary data centers are located in downtown Manhattan — they weren't flooded but ConEd lost power to the area for 5 days, which is a duration we don't plan for. Fortunately our consistent maintenance and testing of backup power systems and our fuel delivery stand-by contracts proved beneficial. We have a full off-site DC 60 miles away but it takes +/- 24 hrs and big $$$ for full activation, so we gambled that Con Ed would get power back on and we stayed put."
"Even when you have fuel contracts, it is not always a guarantee they you will get delivery. One of our sites had to contract companies from down south to get fuel deliveries."
"Had a fuel truck on campus in case of power outage."
"We experienced issues with fuel oil delivery so we are thinking of increasing our capacity on site fuel reserves and we have also increased the amount of fuel oil vendors that we deal with."
"Make sure you have the circuit # from the utility so you can inform them of any issues you are experiencing. Our data center was on a separate circuit from the rest of the hospital."
"Our 3rd production/back-up site is in Queens, where our 24/7 emergency services and call center are located, unfortunately in a Stage 1 flood zone. We had staff and hi-capacity pumps standing by in the basement/switchgear room when flood water began pouring in. Over the course of two hours we literally kept the waters to within an inch of the gear, ready to shut down at a moment's notice."
"We had to bring in employees from other states as the employees who worked in the data centers have significant damage to their homes and their first priority was caring for their family."
"We realized that staff accommodations need to be made with the facility so that the infrastructure that supports the DC supports the staff as well. Surrounding resources were not as reliable as our infrastructure."
"Set up deal with a local gas station to get gas for hospital employees, kept our doors open."
"Initiated an environmental rounds teams that rounded hourly through entire [hospital] facility and reported to HEOC."
"Back up data is important."
"Keep on equipment PM schedules, know equipment specs, and respond to problems quickly."
"Training and cooperation play a very large role in a disaster, the lessons of 9/11 and the blackout are ingrained in our minds."
"We planned and manned the site with maintenance personnel and switched from utility to generator once winds started to increase above 40 miles per hour."
"We were in the middle of a project to find and correct any single point of failures and I think this is what helps us weather the storm."
"We learned that we are just as susceptible to strong hurricanes just like anyone else."
Data Centers: Surviving Hurricane Sandy
Survey: The Effects Of Hurricane Sandy On Critical Facilities
