The blackout that darkened buildings from New York City to Detroit and Toronto served as a vivid reminder that electricity is the lifeblood of safety, security and commerce. Like the problems in California in 2000 and 2001, the August outage exposed the vulnerability of the transmission grid and raised the question of how prepared facilities are if power is lost for an extended period.
Although it is still too early as this issue goes to press to know precisely what went wrong, many lessons from the blackout are already evident. Conversations with facility executives who lived through the August blackout, or experienced the problems in California first-hand, reveal that many steps can be taken to improve response to a major power outage.
Consider evacuation procedures. These have become a high priority since the terrorist attacks of Sept. 11. But the blackout made it clear that occupants need information even before evacuation begins. Without timely access to news sources during the blackout, erroneous rumors arose, resulting in fear and inappropriate action.
One way to distribute information is through a fire notification system that can act as a building-wide public address (PA) system. Under some fire codes, however, such systems may not be used as PA systems merely to convey information. If that sort of limitation exists in an area, action through the local government may be needed. The PA system should be supported by emergency power; otherwise it will be useless during an outage.
To get up-to-date information, one firm routed power from a desktop computer uninterruptible power supply to a small television. Occupants then knew quickly that the outage was widespread and unlikely to end soon, resulting in an early evacuation. In another case, however, employees lacking such information filled in crossword puzzles while waiting for power that did not return.
It might also be useful to have a list of employees who have portable radios or other battery-powered systems — and to be sure that spare batteries are available.
Facility executives should recognize that evacuation instructions related to a power outage are different than those used when a fire is involved. For example, employees should be told to gather their belongings in preparation for an evacuation via the stairwells. And they should be reminded to carry flashlights and wear appropriate shoes, if available. If a PA system is to be used, it may be worthwhile to prerecord messages related specifically to a power outage.
Some facility executives who thought compliance with fire and building codes was sufficient to handle a major power outage discovered that more could have been done in advance to minimize problems. For example, facility executives should determine how children and occupants with disabilities will be evacuated.
Emergency lighting presents another set of issues to consider. Complaints were heard regarding exit signage and corridor emergency lighting that went dark early into the blackout. One way to avoid that problem is to verify the duration of backup battery output in exit, stairwell, corridor and elevator lighting.
Another way to identify problems that might occur during an evacuation is to perform a fire drill using only emergency lighting. Evacuation under those conditions might go very differently than a typical lights-on fire drill.
In many areas, emergency lighting is only required for paths of egress, such as stairwells and corridors, or where large numbers of people congregate, such as exhibit halls. Never assume that the lighting will be sufficient or fully operative, especially in bathrooms, enclosed offices or conference rooms.
As a practical matter, it’s worthwhile to keep a penlight on hand. In New York, some facilities had stored glow sticks that could be used for light both during evacuation and while out on the street. In some cases, building occupants used candles and cigarette lighters as light sources. Many unintentional fires were started during the August outage, and at least one set off a smoke alarm, frightening occupants and confusing building personnel. A better choice was to use light from laptops, cell phones and handheld devices to guide people through the dark.
Seemingly small details loom large during emergencies. For example, facility executives should determine whether fire captains — designated employees responsible for getting people out of offices during fires or drills — are equipped with rechargeable flashlights that are kept in recharge mode. Another point to consider is whether substitute fire captains have been designated to fill in for those who are out of the building during an outage.
The blackout tested another element of emergency plans: backup power. Some diesel generators units ran out of fuel. If fuel suppliers were open and reachable by phone, traffic jams slowed deliveries. Fire codes justifiably limit the amount of diesel fuel stored on site. Rules and inspections for enforcing such limits were stepped up after Sept. 11, when Seven World Trade Center was destroyed by a fire fed by a 40,000-gallon diesel fuel tank.
Some facilities are looking into retrofitting diesel generators to also use natural gas. Many generators can be converted without major overhaul with “fumigated” diesel kits that supplement combustion of diesel fuel with natural gas. Doing so may stretch diesel fuel supply by 400 percent without new tank capacity.
Several emergency generators overheated during the first hour of the blackout. Diesel service firms recommend running emergency generators at least once a month, preferably for an hour, while monitoring temperature. Verify that the unit starts automatically when a voltage drop occurs, instead of needing to be manually started.
Some facilities have added alarms on building automation systems that sound when a generator’s automatic transfer switch activates, indicating a utility voltage drop, which may occur just before an outage or during power restoration. If no automatic transfer switch exists, an alarm indicating low utility voltage can be added.
At least one vendor of distributed generation now offers a form of elevator control software that sequences power to a group of elevators so that each can be brought down, one at a time, and emptied before emergency power is transferred to the next elevator. No elevator is stuck in transit for more than a few minutes.
Power problems can also arise as utility service is being restored. Facility executives may be called upon to reduce load to avoid another blackout during power restoration. If that happens, is there a predefined load triage protocol? Also important to determine is what power startup surge protection exists for computers and other sensitive systems. For example, determine whether there is whole-building surge protection to handle possible voltage spikes when power is restored.
The blackout also revealed a new reason to participate in demand response programs. Facilities in these programs were asked during the first hours after power was restored to limit power use or continue operating generators to support the grid. Whether they had any demand to curtail or generators to run, those enrolled in the programs were notified when the margin between supply and demand was getting tight, tipping them off that a rolling blackout was imminent.
Another utility-related issue is steam. More than 100 U.S. cities are served to varying degrees by central utility steam distribution. Buildings with steam turbines and absorption systems depend on steam for cooling.
During the blackout, New York City’s steam system lost instrumentation power, causing it to shut down and lose pressure. Utility steam was not again available until Sunday night, nearly three days after power had been partially restored. Full steam pressure did not return until Monday morning.
Some facilities with steam-powered cooling — including the American Stock Exchange — were too hot to be occupied Friday afternoon even though the building had power, causing businesses to close early. Once power returned, those with hybrid cooling systems, which include electric chillers, were able to limp along, but doing so increased power demand just as facilities were being asked to cut loads.
In addition to operational and safety concerns, a variety of sticky legal issues are worth looking into in advance.
For example, consider whether an extended outage triggers any “force majeure” clauses in insurance, leases or other contracts. (The term refers to events like riots and acts of God that are beyond the control of the parties involved in the contract.) It’s also worth finding out whether coverage would be affected by any civil unrest that might accompany an outage, as well as what tenants expect with regard to emergency power supplies.
Another point to consider: Insurance coverage could be lost if the facility was ill-prepared for an outage. Nowadays, the cost of one fatality or lawsuit caused by poor planning could easily exceed the cost of preparation.
Facility executives should be aware of commitments organizations have to provide a service or product, such as providing a hotel room, that is stopped in-process. There should be a clear statement on registries, receipts and other documents that defines who owes what to whom, both during an outage and once power is restored. Consider installing noncomputerized systems and providing training to time-stamp damages and expenses incurred during a blackout for use in later claims.
During the week after the blackout, authorities called for short-term power conservation to help restore grid stability. In Cleveland, companies on interruptible rates had their power cut. In Ontario, more than 150,000 federal, provincial and city employees were sent home to conserve power.
The Ontario approach makes sense. If city agencies form a significant part of the utility load, ask that it develop a plan to send nonessential employees home and shut offices to cut demand. Nonessential services and institutions, such as museums, should be part of that plan.
Ontario officials also asked businesses and residences to voluntarily cut power use by 50 percent. However, when some retail stores shut off their outdoor lighting displays but saw others still lit, they turned theirs back on. Emergency blackout power procedures might need to be clarified or codified to make such requests work.
In the New York City area, grid officials ordered a rolling blackout on the second day of the event, but no notification was given to the cities affected. Businesses that had just opened had to close again, and elevators that were running were again stopped mid-floor. Things didn’t settle down again until New York City Mayor Michael Bloomberg threatened a lawsuit, and all agreed some new procedures were needed.
During the California blackouts, utilities and local media set up automated e-mail systems to notify occupants in “service blocks” that defined an outage queue to fairly distribute power. Areas with health and security facilities, such as police stations, hospitals and fire stations, were given priority over zones lacking such facilities.
If the local utility and ISO offer load-curtailment incentives, ask that application forms be posted on their Web sites and that of the city’s emergency preparedness office. Such programs should also be promoted in utility bill inserts and advertising. As more facilities are set up to curtail loads, or turn on emergency generators, utility load managers have more options to limit or avoid outages.
Ask local phone companies, including wireless services, how and for how long their power is backed up.
Request that the city’s emergency preparedness office allow use of the Emergency Notification System, which interrupts radio and TV broadcasts, when an outage is imminent or likely. Doing so could reduce the number of people stuck in stalled elevators or amusement park rides.
Facilities exist to house business operations. Conversations with facility executives revealed a range of tactics to help keep operations going.
No matter how thorough the preparations, no one can anticipate every contingency that might arise in an emergency. But taking action now can eliminate many foreseeable problems and smooth both the immediate response to the emergency and the process of getting the facility back in operation.
Lindsay Audin is president of EnergyWiz, an energy consulting firm based in New York. He is a contributing editor to Building Operating Management.
If Sept. 11 alerted facility executives that emergency response plans need to be ready at a moment’s notice, then Aug. 14 was the moment.
Last month’s power outage tested response plans developed following the terrorist attacks on the World Trade Center and Pentagon. And if early reports are any indication, office buildings fared as well as anybody could expect.
“By and large, we’re thrilled with the response,” says Ian King, who oversees Jones Lang LaSalle’s investment property portfolios.
The most immediate concern in the minutes following the blackout was how to get people out of elevators stuck between floors and ascertaining whether fire and life safety systems were still working, says John Santora, vice president of asset services for Cushman & Wakefield.
Within the 70 million square feet that the firm manages in the New York area, Santora says, seven elevators were stuck. Those lifts were cleared within 20 minutes.
The next step was to evacuate buildings. Santora says as word spread that the cause of the power outage was related to transmission lines and not a terrorist attack, evacuations proceeded orderly. That’s a different scene than when the World Trade Center was attacked.
“After going through that, this power outage was a cakewalk,” says Santora, whose firm saw 20 million square feet of property it oversees damaged as a result of the World Trade Center attack.
The biggest lessons property managers say they learned from the power outage are that phone service and relationships with vendors are critical during crisis situations.
Edward Riguardi, managing director with Jones Lang LaSalle, says although digital phones failed, building managers were still able to communicate through analog phones placed in the management and security office of every building — a lesson learned from Sept. 11.
Santora says his firm used phones when that was possible, but had to revert to physically sending somebody to visit its facilities where phone service was out. He’s now considering using two-way radios for future crises.
The blackout will produce small changes to response plans rather than major overhauls. At Owens Corning World Headquarters, for example, a manual transfer to emergency power will be converted to automatic transfer to prevent even a brief loss of power to the customer service center.
Vendors played a crucial role, too, King says. Aside from needing fuel distributors to make deliveries to keep backup generators running, property managers needed extra security personnel until power was restored. Electronic locks fail in the unlocked position when there is a power failure.
The power outage had a ripple effect on other essential services. Many buildings lost their sanitary water systems during the blackout because pumps had no power.
Incoming water can also be an issue. “If you have a tower feed, you’re in great shape,” says Donald Sposato, director, corporate engineering and technical services, Becton Dickinson. “But if city water is dependent on the power grid, it’s prudent to make inquiries into the level of backup power.”
— by Mike Lobash, executive editor
Several U.S. General Services Administration facilities in Detroit and Cleveland fared well in last month’s blackout to the relief of the facilities’ property managers, who say all their preparations for emergencies proved effective.
“We’ve gone through some real scenarios and drills so we believed we were prepared, and this showed it,” says Laura Marble, property manager at the 670,000-square-foot Theodore Levin U.S. Courthouse in Detroit. Marble’s experience with explosions in the sewer system earlier in the year also helped prepare for the blackout. “It showed us we needed to maintain regular contact with local police and fire officials before, during and after an emergency,” she says.
The outage brought unexpected twists, such as loss of communication. For up to an hour after the blackout, Marble could not contact anyone — inside or outside the building. Cell phones and the facility’s Nextel radio links were knocked out. For a quarter hour or more, Marble didn’t know the blackout involved more than her building.
To find mechanics and contractors in the building, she used CCTV cameras. She also used a new fire alarm system that included a public-announcement function.
Luck was on Greg Wade’s side, property manager for the 1.4-million-square-foot Anthony J. Celebrezze Building in downtown Cleveland. While his building escaped the water problems facing Cleveland during the blackout, he would’ve been “in a world of hurt” had it not been for the recent upgrade of emergency power and elevator systems.
“That upgrade and testing of the various components opened our eyes to what was on the system and what wasn’t and how it will work when needed,” he says. For example, tests found key emergency lights and the garage doors weren’t on the old backup power circuits. “I heard people in other buildings were having a problem getting out of the parking facility because garage doors weren’t working.”
With all his digital communication also inoperable, Munir Muhammad, property manager of the 1-million-square-foot McNamara Federal Building in Detroit, turned to hoofing it and an old analog land line to contact agency heads. But because most of the key phone numbers were on his computer, he had trouble reaching them. “First thing I did when I could was to run out a hard copy of all those phone numbers: office, cell and home numbers,” he says.
But communication didn’t turn out to be Muhammad’s biggest headache — loss of water supply was. Despite a successful evacuation, there were still dozens of security personnel and contractors in the building using the restrooms. “By Friday the smell told us we needed some portable facilities,” Muhammad says.
— by David Kozlowski, senior editor
For telecommunications companies like AT&T, loss of utility power in a blackout is only half of the problem. “The first thing people want to do in an emergency is pick up the phone,” says Alan Abrahamson, real estate operations director. The additional calls strain the system just when it is at its most vulnerable.
AT&T maintained service throughout the blackout. But the company’s mission-critical facilities and strategies were tested.
The company has portable generators with quick-connect capability that can be trucked to critical locations if an on-site unit fails. But on Aug. 14, the backup plan needed a back-up.
One generator had to be moved into New York City over the George Washington Bridge. But traffic was at a standstill. “The truck got stuck for hours before a police escort got it through,” says Abrahamson. “Fortunately, we had battery life to carry us through.”
Most data center and telecommunications spaces remained up and running. But some reportedly did go down.
One Achilles heel was equipment failure. “Given the length of the blackout, the likelihood of having a component failure is fairly high,” says Peter Gross, chief technology officer, EYP Mission Critical Facilities.
To minimize the risk of failure, continuous maintenance is important. So is ongoing testing, like running generators on full load. At AT&T, says Abrahamson, “tests are done on a regular basis. It’s a very rigorous schedule.”
Human action — or inaction — can undermine reliability. In an emergency, operators can be overwhelmed by alarms. “They have to be able to discern which are critical and respond properly,” Gross says. Training is crucial.
The outage exposed the way “equipment creep” threatens reliability, says Rob Friedel, senior vice president, Syska Hennessy Group. Equipment creep occurs as more power-hungry, heat-generating gear is crammed into space not designed for the added load. “That cut into backup time and system reliability,” Friedel says. “People thought they were well-covered but they weren’t.”
Even when the infrastructure performed well, the length of the outage raised questions. Consider a data center located within a larger facility. “You have a little box within a big box,” says Donald Sposato, director, corporate engineering and technical services, Becton Dickinson. The data center cooling system may be designed to maintain 70 F, while the building itself is kept at 74 F. “If the temperature of the big box starts to creep up, the little box has to work harder,” Sposato says. A long outage on a very hot day could strain the capacity of the data center cooling system.
— by Edward Sullivan, editor
