Fred Hutchinson Cancer Research Center, a world leader in cancer research, employs a strategy of expanding its emergency power system as needed instead of over-sizing in anticipation of future needs.
August 3, 2015 - Power & Communication
It's a conundrum faced by facilities managers and engineers all too often — planning for infrastructure growth. Should an infrastructure larger than needed in the near term be built for anticipated long-term growth? Or, should it be built to satisfy only near-term demand, with more infrastructure added later as needed? ASCO Power Technologies works with facilities managers who come on both sides of that fence.
Operating engineer Mark Hungerford, for example, has wrestled with the conundrum for 15 years as the operating engineer for a medical research and healthcare campus that's grown six-fold since 1991 and now covers 15 acres and employs 4000 people.
"We generally follow the model that was used for the first new building; meaning we start with a basic, robust infrastructure of high quality equipment that will accommodate expansion, and grow out from there, when and where necessary," he says. That first building was the beginning of the Fred Hutchinson Cancer Research Center, a world leader in research to prevent, detect and treat cancer and other life-threatening diseases.
Later, in 2001, the Center joined with two other leading healthcare institutions in the city to create the Seattle Cancer Care Alliance, a world-class cancer treatment clinic that's also part of the campus. The campus itself is in the Denny Triangle area of downtown Seattle, a revitalized urban neighborhood of extremely rapid growth catering to the biotech industry.
While the term 'infrastructure' covers a lot of ground, Hungerford cites the campus's emergency power system as an example of adding equipment as needed: "We start by determining what loads should be on emergency power on the day the doors open, then build off that backbone as our needs and experiences dictate."
As a case in point, the Center recently acquired an existing 177,000 sq. ft. building near the campus. Hungerford, as an evaluator of electrical systems for new Center buildings and construction expansions, reviewed the existing infrastructure and helped determine emergency power requirements that would provide the level of reliability necessary for the operations it would house in the near, mid, and long-term.
This approach has served the campus well. Today, it's recognized for its reliable and right-sized infrastructure.
"It's All About Reliability..."
Hungerford emphasizes the point: "It’s all about reliability. Everything, especially our data centers, are built to never go down."
That's because campus operations scream 'mission critical!' Data centers totaling 18,000 sq. ft. store colossal volumes of information produced by more than 200 research labs and advanced imaging facilities, cell monitoring and manufacturing operations, and specialized tools that analyze and sequence DNA and RNA.
The data centers, one of them a new $9 million facility, also store information relating to patient prescriptions and medical histories, employee payroll, back up data for personal computers and other electronic devices and, of course, e-mail.
Power reliability also is critical for more than 900 freezers that maintain samples at sub-zero temperatures, sometimes for decades. Freezer contents include completed research, experimental results, and patient samples: truly irreplaceable items that can represent years of scientific effort.
At the Seattle Cancer Care Alliance, power reliability is critical to help ensure smooth patient flow. During a single visit, patients may see an oncologist, get medications from the pharmacy, and perhaps undergo an MRI or CAT scan. Due to a heavy daily volume of patients and caregivers through the building, it all needs to run like clockwork.
A power outage, however brief, upsets that flow, quickly bottlenecking procedures. The necessary rescheduling of appointments, and the restarting of sophisticated electronic equipment, are activities that are disruptive, time consuming, and expensive.
A Powerful Infrastructure
The emergency power infrastructure that supports campus operations is based on a distributed topology with seven stationary, automatically-started, diesel-powered generators and twenty-two low voltage ASCO Automatic Transfer Switches, located throughout the campus. One large portable generator with 1600 feet of 400 ampere soft cable on deployment reels provides supplemental emergency power, via permanently installed quick-connect points, if necessary.
The eight generators develop nine megawatts (MW) of power at 480V. Two of the units are located in the recently acquired 177,000 sq. ft. addition, and operate in parallel. For now, one is redundant, but ongoing load growth will eventually consume its capacity as well. The other campus equipment is not paralleled due to the distance between buildings; i.e. each building contains its own stand-alone generator and ancillary emergency power systems.
"Our transfer switches range in size from two-hundred to three-thousand amperes," Hungerford explains. "They connect life safety, critical and essential loads to either normal or emergency power sources, as external conditions or our operations require."
Most transfer switches operate in a delayed transition mode, and all but a few have bypass-isolation capability. As part of ongoing work to boost reliability, Hungerford employed the manufacturer's service group, which has almost as much experience with the campus' equipment as he, and a qualified electrical contractor, to convert two transfer switches to isolation-bypass devices.
One isolation-bypass conversion was an 800 amp, 208V transfer switch serving a central freezer facility; included in the upgrade was a utility-sanctioned, closed transition mode. The other isolation-bypass conversion was a 1000 amp 480V switch for the Seattle Cancer Care Alliance. In both cases, Hungerford specified the highest seismic rating available. For these and five additional transfer switches undergoing upgrades, he specified new, advanced microprocessor controllers that log switch activity, and allow programming without opening switch enclosure doors, which was a safety concern.
To maintain load power during switch upgrades or replacements, Hungerford routes a temporary soft cable protected by circuit breakers around the affected transfer switch and to the load. Facilities personnel meter the cable to avoid overloading. Wherever necessary, housekeeping pads are enlarged to accommodate the larger isolation-bypass switch footprint, as part of the upgrade.
Planning ahead, Hungerford recently submitted a budget request to replace an additional 800 and 2,000 amp 480V transfer switch with bypass-isolation types. In general, space and working distance requirements within electrical switchgear rooms are very tight, so accurate manufacturer's drawings of their products and multiple options for cable entry and exit can be essential. "An inch or two can make a difference to a retrofit project," he says.
Regarding brands and manufacturers, Hungerford's advice is, "With critical equipment like transfer switches, once you have found a brand you know to be reliable, and with which you are familiar operationally, stick with it. A competitive purchase price is important, of course, but never lose sight of long-term costs. There is great value in standardization, especially when it comes to operator safety.”
For the data centers and other campus operations that must see no interruption of power, whatsoever, an array of 175 uninterruptible power systems, ranging up to 750 KVA in capacity, provide electricity until the generators pick up the load.
On the Center campus, emergency power generally serves less than half of each building's total load. The two generators for the new 177,000 sq. ft. addition back up more than half, however, which may be the start of a trend on the campus. The litmus test Hungerford uses to determine whether a load should be backed up is, of course, how long can equipment be off without negative consequences?
The answer for research work is "not long." Comparative medicine operations requiring tight airflow and temperature control; the manufacture of medicines, from live cells and cultures, needed by a patient on a specific date; and any research process utilizing sophisticated electronics, all need reliable power.
Power Distribution Design
Campus power distribution begins with a combination of three to four incoming utility 28 kV feeders, depending on the city’s dynamic load profile. Inoperable utility sources, whether due to unplanned power outages or planned-for construction, can be bypassed or fed alternately, as necessary. Two large Seattle City Light substations, four miles apart, supply power to the principle feeders.
Twelve in-building 28,000/480V transformers provide a total of 30 MW of power to Center research, clinical, and administrative operations. In critical research buildings, main-tie-main switchboards and dual utility transformers provide redundancy at up to 4000 amps 480V.
The campus stores 24,500 gallons of diesel fuel on site for emergency generator and boiler operations. If permanently-installed fuel replenishment systems are inoperable, a truck with a tank and pump are available to move fuel.
Maintaining proper selective coordination, too, is another important consideration to Hungerford and his facility team. "In a place the size of this campus, and due to the continuous alterations and construction, it’s conceivable to install a breaker that adversely affects selective coordination of overcurrent devices," Hungerford says.
"A misapplied breaker can fall short of the required short circuit rating, too, which is a mistake that can have disastrous results on people and equipment. For this reason, new work includes a review of circuit breaker or fuse coordination and AIC ratings."
Testing and Advocating for Superior Equipment and Features
Routine emergency system testing, too, gets close scrutiny. A single monitoring system tells Hungerford what's happening throughout the entire power distribution infrastructure. But, he prefers “feet on the ground” for monthly mandatory testing. Facility personnel initiate tests at each transfer switch and record readings that will be used to produce reports for both the Joint Commission and the Seattle fire marshal.
"Being at the equipment and hearing what you can hear, smelling what you can smell, etcetera, is important," he says. “Sometime it’s a matter of picking up on a change through your basic senses and, hopefully, discovering a problem before it becomes much larger and worrisome.”
Hungerford's expertise with the growing campus has built 'intellectual capital’ that stands him in good stead when he's advocating for electrical system improvements, or when tight construction budgets may push to value engineer out the bigger generator, the better UPS, or the bypass-isolation feature on a transfer switch.
"When we explain it on a level that shows direct impact, our arguments for the better product or superior features are usually accepted," he says. “No one is eager to go on record as saying we can’t afford this or that, and then, a couple of power outages later, find that what we advocated for is a necessity. And the price to add it later is always much higher, and in other ways painful, of course.”
Proper foresight and design of emergency power systems is all a part of solving the conundrum of building and maintaining a reliable infrastructure for a rapidly expanding, mission critical medical campus. For Hungerford and his team, it's all in a day's work.
