When a huge storm hit a stand-alone surgical center located near the coastal city of Moorehead, N.C., a power outage put backup generators and uninterruptible power supply (UPS) systems to the test.
Because the surgical center is so close to the storm-prone coastline, the campus includes an employee shelter to accommodate extended stays. To support the extra space, the facility’s generator was larger than typical installations for stand-alone surgical centers of that size.
It also was more costly to test than typical installations. The facility’s somewhat remote location made fuel delivery costs significantly more expensive than in-town deliveries.
Unfortunately, soon after the service contract expired, regular testing ended. When the storm hit and the generator was pressed into action, its performance was less than stellar.
“It ran rough. They wanted to get rid of it all,” says Mike Daugird, project electrical engineer for Marshall Erdman and Associates, who specializes in backup power systems for the health care industry.
But the problem wasn’t the equipment. It was the lack of testing.
Even though that generator didn’t fail under pressure, it could have. The failure of a generator or UPS system is all too common, even in facilities designed for high reliability, says Rajan Battish, principal of RTKL.
In one hospital data center, a solenoid valve between the day fuel tank and the main fuel tank failed. When power went out, the day tank had only 90 minutes worth of fuel. Once that was gone, the engine shut down. The UPS system downstream of the failed generator tried to carry the data center load. But due to lack of proper maintenance, some cells in the UPS batteries had failed. The UPS dropped the load, bringing down the data center.
Critical facilities are generally designed to avoid single points of failure. But a combination of events can bring down even well-designed facilities.
Battish cites a data center with redundant UPS systems. When batteries needed to be replaced on the A side of the UPS system, generators were started up to handle the load. But a momentary glitch in the utility power supply overloaded the generators during a closed transition operation. When they shut down, the dual-corded servers switched to the B side of the UPS system. Both the A and B sides were designed to handle 40 percent of the facility load. But the UPS A/B redundancy hadn’t been managed properly, says Battish. Both sides were loaded at 60 percent. As a result, the power strips in the racks were overloaded. They tripped off, bringing the servers down.
A backup generator is nothing more than a diesel engine, similar to what you’d find under the hood of a semi truck. If it’s not started from time to time, there is no assurance it will turn over when needed.
UPS systems fail for the same reason a generator’s battery fails: They’re ignored. A UPS system is simply a massive bank of batteries. Typically these batteries fail because they have exceeded their lifespan and aren’t replaced.
There are plenty of reasons generators fail. For example, if the generator runs for a while, then shuts down, there may be an electrical failure within the backup system. Or the generator may have gotten too hot. “That can be the result of not enough water in the radiator, or it’s not sized for hot weather,” says Christopher Johnston, vice president with Syska Hennessy Group.
Generators also will spit and sputter when there’s a problem with the fuel delivery system. If the pump on the tank isn’t backed up with some sort of secondary system, primary system failure means there’s no way to get fuel from the tank to the main generator. That means the system is down for awhile. That kind of problem can be ferreted out during routine testing and corrected long before the generator is pressed into service.
If the generator runs well for a while, then freezes, it’s probably out of oil. “Diesel generators burn oil,” Johnston says. “If the diesel generator is required to run a long period of time, in a hurricane, for instance, eventually you have to change the oil.”
Oil can’t be added while the engine is running, nor can the filters be changed. Because this is a known failure waiting to happen, provisions must be in place to keep the power going during this maintenance.
One company learned that the hard way. “The company had to run diesel generators for a couple of weeks during the summer after a hurricane,” says Johnston. “One generator failed. That left two generators. One generator had to be shut down to add oil. The other one was so heavily loaded that it began to overheat, and it failed. The data center was lost.”
One way to eliminate failures is to design performance mistakes out of the system, says Gerry Murray, engineering design principal with KlingStubbins.
In mission critical facilities, he says, the design goal is to prevent anticipated failure modes. For example, generators generally require vents. If the venting system fails, it should be designed to fail in the open position.
The equipment also should be housed in a space where it’s not subject to flooding or water run off during storms. “We hear stories all the time of generators in basements which flood or don’t have very good ventilation,” Murray says.
Maintenance features designed into the building can improve performance as well. For example, permanent on-site load banks allow for regular testing without any impact on the critical load.
“In the ideal world, you would set up the load banks to load the UPS and the generator,” Murray says. “You design for maintenance.” For example, an onsite load bank eliminates a big disincentive to regular testing: The need to rent one, which can cost thousands of dollars.
Systems that have multiple generators can be programmed to include a feature that rolls through the load testing down the line, sequentially.
Murray suggests that facility executives consider their fuel needs across their most likely scenarios. Some mission-critical facilities require multiple fuel tanks for extended operation. For example, a data center located along a hurricane-prone seaboard might need to plan for days of extended operation.
Because bad fuel is a major cause of failure, multiple fuel tanks — including a dedicated delivery tank — would allow facility executives to test the fuel before it’s pumped into the central storage tank. A secondary fuel tank would keep the system online should the primary tank fail.
Another design consideration is the level of redundancy the system needs. One way to achieve redundancy is to join battery systems together through isolation diodes that allow the control system to be powered from more than one source. “We generally put one more station battery over and above what’s required so you have multiple sources of power, which is crucial in starting the generators,” Murray says.
Meanwhile, a programmable logic controller (PLC) automatically moves the system to the UPS when the main voltage from the grid drops below a certain level. That approach prevents the problem of the generator not being instructed to start during a brown out.
Once the UPS kicks in, it notifies the generator system that the building is running on battery. This gives the generator time to ramp up to full voltage before the UPS starts to transfer the building’s load.
If the generator is not allowed to ramp up to full voltage before the load is transferred, the UPS will take back the load, causing the generator to stand down. Sometimes when the load moves from one system to another, it causes lights to flicker, or even worse, internal building systems to reboot. These systems will continue to reboot if the load shuffles between the UPS and the generator.
“Consider how equipment will be operated in the real world,” Murray says. “You need to pick a specific strategy that will help start the generator system.”
Almost all failures can be predicted and corrected with routine testing and maintenance.
When any new back-up system is first installed, it needs to be tested and commissioned to make sure it operates correctly before it goes online. After that, the system should be tested once a month under load for at least an hour. If the system has more than one generator, each generator should be tested individually.
Once a year, the whole system should be run under full load for at least eight hours to mimic how the building responds under these conditions. But Battish says eight hours is a minimum. “You really need to go beyond eight hours,” he says. For a system designed to run for 72 hours, he recommends running the generator at least 24 hours at least once a year.
The generators should be run under load, rather than simply letting them idle. Facility executives have three options. One is to use a resistive load bank, which is essentially a large heater. The problem with that approach is that a building load isn’t just resistive.
A better idea is to use a reactive load bank. That provides a better signature of the power distribution system. But even that doesn’t provide a true picture of the building load.
The best approach, says Battish, is to run the generators and UPS system under the full building load. That way, the facility executive gets to test the system with all the noise, harmonics and inrush currents that it would be exposed to in the event of an outage.
As the system runs, vital signs such as how well it starts, oil pressure and exhaust temperature can be checked. How the generator responds under pressure and how the building responds to the generator can be determined. This information can be used to develop facility management procedures for crisis situations, Daugird says.
Testing also will point out any flaws in the system. For example, problems with the fuel system can be discovered and corrected.
The alternator needs to be checked annually to make sure it’s in good condition, and the output circuit breaker needs to be inspected every five years or so.
Daugird also recommends that the generator’s heater be checked regularly to ensure it’s turned on. The generator engine must be heated to about 110 degrees F to take over the load without thermal shock.
In the name of energy savings, these heaters sometimes are turned off. As a result, generators that must crank up in the dead of winter in cold climates won’t start because thermal shock causes the generator to stall as it kicks on.
Although generator testing is common sense for all facilities, it is mandatory for facility executives in health care organizations. All health care facilities are obligated to comply with National Fire Protection Association code for generator testing. The Joint Commission on the Accreditation of Health Care Organizations also has requirements that apply to most health care facilities.
Air quality problems can complicate frequent testing. In some states — for example, California — generator testing must be scheduled and approved in advance. It also must be in compliance with air quality standards.
For someone not schooled in building operations, testing costs can be a shock to the pocket book. Generator-grade fuel is lower in sulphur and more expensive than over-the-road fuel from the corner gas station. A 2-megawatt diesel generator burns 120 gallons her hour.
UPS systems typically are tested on a quarterly basis, but there are some facility executives who have problems arranging for testing time. UPS systems are the first line of defense should power go out.
“If you want to test a generator that’s one thing, but testing the UPS is a whole different set of circumstances,” Murray says.
Maintaining UPS systems can be tricky, particularly if the battery label says “maintenance-free.” Maintenance-free really means a valve-regulated battery that should last for about five years. Every time a battery is used, however, its life degrades. If the battery pulls the building through multiple power outages, its life span shortens. If it’s not used after five years, it might work. Or it might not.
If getting the longest-life battery is the biggest concern, Daugird recommends a flooded battery that is maintained monthly. These batteries do require more space — a dedicated 12-by-12-foot vented room to eliminate toxic gases — and are substantially more expensive than valve-regulated batteries. Flooded batteries might be used in a data center when downtime must be kept to a minimum.
Generally the UPS system should be tested twice a year with load, so that meters can be checked, diagnostics run, and thermal scans conducted to look for hot spots. Six months later, it should undergo more comprehensive testing that involves taking the system offline and putting the building’s load on bypass. The system should be run four to six hours to shake out any problems.
Some facilities check all equipment daily or weekly to ensure that alarms are working and that the system is free of trash, leaves and other debris. They look at coolant, oil levels, battery charges and the belts.
The fuel itself also should be checked — particularly if the generator isn’t being tested. “Bacteria love diesel fuel,” Johnston says. “If bacteria gets in the tank, it forms a layer of sludge on the bottom of the tank. When the fuel pump starts pulling sludge, it clogs the engine.”
Routinely tested generators are less likely to suffer this problem.
“What you do daily, weekly and monthly, every six months and yearly should be customized to the facility,” Murray says. “If you get a lot of outages, then your generators are being tested. If not, monthly or quarterly testing is even more important. Most of the horror stories come from facilities that don’t have that level of inspection and maintenance.”
There are implied risks with testing, but the risk of going into an outage with an untested system is greater, these experts say. Systems that are regularly used should perform more reliably than systems that aren’t. Untested systems are more likely to suffer failures when a real outage occurs.
Given the risks of not testing, why aren’t generators and UPS systems put through their paces more often? One big reason is the risk that inexperienced personnel conducting the tests could make a mistake that triggers a failure.
“It’s a serious risk,” says Battish. “No one wants to be the one who brings the site down.”
Cost is another obstacle. “Many times, what causes generators to fail is money,” Daugird says. “Maybe someone tried saving on the upfront costs of a higher quality system or on maintenance. Something is skipped to save money.”
Part of the cost of maintaining backup power systems is related to people. An in-house staff qualified to maintain a critical facility 24/7 requires a significant investment. Another option is to contract for system testing and maintenance. These contracts can include emergency service, parts rental and fuel and delivery charges. When fuel is included, customers usually have to pay for it whether they use it or not. Because they have to pay fuel costs anyway, this encourages scheduled testing. Service contracts are expensive, up to $40,000 per year.
Is that too high a price? That may not be the right question. It may be better to ask, “What is the cost if the system fails?”
Lynn Proctor Windle, a contributing editor for Building Operating Management, is a freelance writer who has written extensively about real estate.
