Introduction
Imagine if an automobile was designed to operate at a maximum speed of 50 mph and upon reaching 100,000 miles, it was required to speed up to 100 mph, 7 days per week, 24 hours per day, forever. Conceptually, this is what has occurred to the demand for power within the last two decades.

Much of society has been oblivious to the inadequacies of our current electrical system, but the Blackout of 2003 served as an international “wake-up call” to the vulnerability of the power distribution system and the need for infrastructure upgrades. Today’s electrical distribution system was established over 50 years ago and founded on the premise of transmitting power to limited analog devices. Revolutionary changes within the past two decades molded society into a 7x24 digital world. Digital technology penetrated into every aspect of society and power demands soared beyond levels anyone could have predicted 50 years ago.

As the demand for reliable and continuous operations continues to skyrocket, the facility operators and the core of buildings’ electrical and mechanical infrastructure seldom change. Unfortunately, the nation’s power generation and electromechanical-based electric grid has not been able to keep pace with the escalating demand for reliable, conditioned power required by today’s digital world. The national electrical transmission network and electrical distribution systems in many existing buildings were not designed to handle the burden of today’s high-tech society.

United States electrical consumption has nearly doubled while investments into the US transmission grid has been reduced by more than half in the past 25 years. (Figure 1) Additionally, we only expect to see continued technological advancements in the future, further exhausting antiquated electrical systems.

Figure 1: Transmission Grid Investment vs. Energy Consumption¹

Further complicating the problem, electric generation in many states has evolved from a stable, stringently regulated, public industry to a competitive, erratic business. Cost cutting procedures became prevalent in states experiencing electric deregulation while the demand for increased uptime continued to grow. As a result, power quality and reliability has and will continue to suffer, only further jeopardizing the stability and dependability of the entire national power grid. As we have learned from the Blackout of 2003, a single isolated failure at any location in electrical distribution system can domino into a nationwide crisis along the intertwined and complex power grid network.

In the past, a power outage simply produced lost work hours and the associated opportunity cost. Today, even a second long power glitch can wreak havoc, causing computer crashes, a tremendous loss of costly data, and damage to online business transactions. Disruption to public safety facilities, such as hospitals, government defense agencies, and air traffic control centers, may even cost human lives. Fifty million people experienced the aggravation and associated costs of a powerless life during the Blackout of 2003. The 29-hour blackout in New York City alone is estimated to cost its economy over 1 billion dollars.²

Proper facility management and maintenance have become of utmost importance and concern for all types of corporations and institutions today. Managers need to adopt new management and maintenance techniques to meet the growing demands with diminishing resources. The key to surviving in the rigorous and high-tech modern business environment is becoming informed of the power risks and being active in the measures necessary to ensure the level of reliability necessary to sustain your business operations.

Transformation into a Digital Society

There was a time when most corporations and institutions operated from 9 to 5 and data centers operated with batch punch cards. In the global market today, all varieties of corporations and institutions have plunged into the Information Age to do business 24 hours a day, everyday of the year. This type of demand requires high levels of reliability and flexibility, whether the companies are large Fortune 1000 corporations or small companies serving global customers. Whatever the type of industry, most organizations are realizing that a 24-hour a day operation is imperative to their business.

The Mission Critical Facilities Engineering field has evolved significantly from standard office build-outs that once included minimal electrical loads such as typewriters, calculators, and task lighting. Mainframe computers of twenty years ago are now powerful personal computers located on desktops in every office environment throughout the world. Devices such as cell phones, PDAs, and ATMs, have evolved from commodities into everyday items. There is a universal reliance on electrical power in the functioning of our daily lives and the driving force of businesses everywhere.

Information drives business and business creates information. The most abundant and readily available source of information is the Internet, which has become as much a household name as a television. The number of worldwide Internet users surpassed 530 million by the end of 2001 and is expected to reach over 1 billion by the end of 2005.³ Final 2002 e-commerce sales were nearly $80 billion and the market is predicted to flourish into the trillions of dollars range in coming years. Many companies and a colossal sum of money rest at the mercy of the mission critical facilities sustaining them.⁴

Power protection equipment today must be able to handle a dangerously wide variety of power disturbances that can be generated anywhere along growing networks of servers, routers, gateways, bridges, and other sensitive electronic components. At the heart of almost all business lines today is extremely vulnerable computer hardware that absolutely cannot tolerate an interruption in power for more than ½ cycle or 8 milliseconds. As a result of this low tolerance level, much attention has been devoted to appropriate power conditioning and management, as well as the associated maintenance, diligent documentation and training for facility operations.

Evolution of Mission Critical Facilities Engineering

If there was a problem with downtime in the past, it was usually a result of computer hardware or software failure. As technology improved, Information Services began to design their hardware and software systems with increased reliability and redundancy. Today, scheduled hardware or software upgrades are the most likely causes for non-power-related computer downtime; however, the computer applications are normally launched on back-up or mirror sites during upgrade periods. As a result of Information Services’ efforts, the computer systems have become more reliable than the electrical and mechanical infrastructure that support them.

Nowadays, downtime is caused primarily by poor power quality, power distribution disruptions, electrical grid malfunctions, environmental disturbances, such as lightning, or even electrostatic discharge (ESD). Minimizing these failures is the responsibility of the modern-day Facilities Manager. The modern-day Facilities Manager has to meet unprecedented demands. Today’s data centers and mission critical facilities need more than conditioned power; they require computer grade electricity at all times. While the technology is available, electrical equipment upgrades have not been at pace with the growing demands.

It is imperative that the facilities engineering department be given the tools, capital funds, and human resources necessary to implement the correct preventative maintenance programs with the support of the board of directors. Unfortunately, to date, the Facilities Manager has often been given limited resources and support to meet the appropriate reliability needs of the business. Senior management must understand the direct correlation between reliability and a positive bottom line. Once management offers Facilities Engineers the necessary tools and capital resources, the building infrastructure may finally begin to concur with the necessary level of demand and reliability. Until then, the Facilities Manager cannot protect against a wide variety of hazards that may paralyze the business at any given time.

Corporations and institutions need to retain a competent Facilities Engineering staff. The electrical distribution systems in mission critical facilities today are so complex that it could take a minimum of one year to fully understand and integrate the systems from a technical, operational, and administrative standpoint. The Facilities Manager has become one of the most vital employees in any given office.

As New York City Mayor Bloomberg mentioned in a radio address during the Blackout of 2003, it is crucial that all major corporations contain a well-trained facilities staff prepared for all types of emergencies. It was these Engineers who worked overtime during the blackout and minimized financial damage to mission critical businesses and sustained operations of hospitals, air traffic controllers, and radio transmission, and utilities to ensure public safety.

Risk Tolerance and Growing Cost of Downtime
In order to design a building with the appropriate level of reliability, a company first needs to assess the cost of downtime and determine their associated risk tolerance. Downtime can no longer be equated to power availability and cannot be measured in terms of “one nine – six nines” (90% - 99.9999%), as recovery time is now a significant component of downtime. Today, recovery time is typically many times longer than utility outages as operations have become much more intricate and complex. Is a 32 second facilities outage really only 32 seconds? Is it perhaps 2 hours or 2 days? The real question is how long does it take to fully recover from the 32 second outage and return to normal operational status?

Facility engineers and senior management need to evaluate the cost of operating with obsolete electrical distribution systems and the associated risk of an outage. When the potential for such losses exists, serious capital expenditures to upgrade the electrical distribution system are monetarily justified by senior management. The cost of downtime in vast industries has expanded tremendously in recent years, as business has become completely computer dependent and the systems have become increasingly complex. (Figure 2)

Imagine you are the manager responsible for a major data center that provides approval of checks and other online electronic transactions for American Express, MasterCard, and Visa. On the biggest shopping day of the year, the day after Thanksgiving, you find out that the data center has lost its utility service. Your first reaction is that the data center has a UPS and standby generator so there is no problem, however, the standby generator has not started due to a fuel problem and the data center will shutdown in 15 minutes, which is the amount of time the UPS system has in battery power at full load. The penalty for not being proactive is loss of revenue, potentially major clients, and, if the problem is large enough, your business could be at risk of financially collapsing.

The above problem could have been identified and prevented if the standby generator was exercised every week for 30 minutes. Precautionary protection is often worth the extra expense. There are about three times as many UPS systems in use today than there were 10 years ago, and many more companies are still discovering their worth after a power line disturbance takes critical equipment off-line and data is lost. Do you want the electrical outage to be scheduled or unscheduled? Serious facilities engineers use comprehensive preventive maintenance procedures to avoid being caught off-guard.

Many companies do not consider installing back-up equipment until after an incident has already occurred. During the months following the Blackout of 2003, the industry experienced an increase number of requests for the installation of UPS systems and Standby Generators. Small and large businesses alike learned how susceptible they are to power disturbances and the associated costs of not being prepared. Some businesses that are not typically considered “mission critical” learned that they cannot afford to be unprotected during a power outage. For example, the Blackout of 2003 destroyed 250 million dollars of perishable food in New York City alone.⁶ Businesses of every type everywhere are reassessing their level of risk tolerance and cost of downtime.

Electric Utility Deregulation
Within the past few years, some states have seen the deregulation, or restructuring, of many electric utilities. (Figure 3) We have also seen the emergence of retail wheeling, or the process of delivering energy supplied by one party over a transmission system owned and controlled by another party. While deregulation and retail wheeling are supposed to provide more choices to the consumer, there are no signs that this will equate to greater power quality and reliability; in fact, evidence shows that power quality has been and will continue to be adversely affected. Changes brought about by this type of restructuring only increase the importance of the facility manager.

Deregulation has already taken its toll as numerous power reliability events took place during the summer of 1999 in New York, Chicago, New England, Mid-Atlantic States and South-Central States and the Blackout of 2003. These regions experienced severe voltage fluctuations and power outages during peak usage, as utilities were not prepared to meet excessive demands and react to emergency conditions.

As the electric utility industry has transformed into a competitive market, cost cutting strategies have been adopted and less resources have been allocated for reliability measures. A January 2000 Department of Energy (DOE) study of the events during the summer of 1999 stated that electric utility deregulation has caused significant degradation in the North American power grid reliability during peak usage periods. The DOE cited aging infrastructure and increased demand for power as the primary causes of stress on the transmission and distribution system that led to the interruption of service. The study concluded that, “State and federal regulatory policies are not providing adequate incentives for utilities to maintain and upgrade facilities to provide an acceptable level of reliability.”

Without mandated supervision from regulatory agencies, electric utilities are encountering difficulty in balancing reliability and price to compete in a deregulated market. Most competitive utilities are attempting to run operations as close to maximum capacity as possible with minimal or no spinning reserves (readily available idling generation equipment) and standby generation capacity. Twenty years ago, however, it was likely to see utilities with spinning reserves in the range of 25% of maximum capacity. Today, utilities are streamlining costs and operating with considerably less spinning reserves than in the past, thus increasing the probability of outages and brownouts, as well as the need for standby power systems. (Figure 4)

In the past few summers, California has also experienced a severe energy crisis as a direct result of the transition to a competitive electricity marketplace. In the past, state regulators and local utilities were responsible for ensuring an adequate and reliable energy supply; however, in a deregulated market, no organization holds such responsibility. The availability of energy is solely based on market pressures, and with a rate freeze on utility service, there is little incentive for customers to limit energy consumption, even during a time of crisis. Californian utilities were left with no resources, nor the incentives, to attempt to meet booming demand as seen in the past few summers.

The DOE predicts that numerous power reliability events will continue to emerge until significant measures are taken to improve reliability of competitive retail utilities. In order to compensate for recent power degradation, some have suggested that a government agency be established to act as a supervisory organization to ensure that all competing utilities are supplying a suitable level of reliability. Others believe that states need to regress to regulated utilities in order to furnish the level of reliability necessary in today’s 7x24 digital society.

The Aftermath of the Blackout of 2003: What do we do next? It is quite apparent that our existing electrical grid is an archaic and inadequate system that does not effectively meet the reliability needs of today’s digital society. The electrical grid is a sprawling network of over 235,000 miles of cables, at times carrying 230,000 volts or more with the backbone of our economy and safety resting on its availability. While the complexity of the system does allow for redundancy, the Blackout of 2003 demonstrated that failures are quickly augmented into massive catastrophes along this interconnected network and how quickly our country can be crippled when power is interrupted.

While engineers have always been conscious of the threat, power outages like the Blackout of 2003 are often required to remind the general public just how vulnerable and dependent everyone is upon our obsolete electrical grid. Our decision makers need to ensure that we are protected and prepared so that a crisis like this never happens again. Many businesses discovered the cost of a blackout of this magnitude and in the process also realized just how invincibility we were as we quickly vanished into darkness.

In the aftermath of the Blackout of 2003 there was a resurgence of national awareness for the need to renovate the current power grid infrastructure. The lives of over 50 million people came to halt that Thursday afternoon and billions of dollars of business was lost. The voices of dissatisfied citizens and business owners have and will continue to be heard on the international platform and utilities must respond.

With unforeseen public support and the abrupt emergence of the electric grid into the political arena, a much larger percentage of the local, state, federal, and private sector budgets will likely be allocated to the reconstruction of the current electric distribution system. Over the next 20-30 years, the government and privately owned utilities will have the opportunity to rectify many of the drawbacks inherent in the current electrical distribution system. In addition to redesigning the means of transmission, the utilities will have the opportunity to expand capacity with new electric generation systems to meet the growing demands of years to come.

During the development of increased generating capacity, national reconsideration of the generation sector could encourage further use of environmentally friendly or “green” power. Photovoltaics, fuel cells, wind power, and geothermal energy are common examples of recent attempts at green power. These systems are considerably more environmentally friendly than existing fossil fuel electric generation. A movement towards green power may help decelerate the progression of global warming and the destructive influence of fossil fuels from emissions of greenhouse gases. While some of these systems have sporadically appeared throughout the world, green power is far from mainstream, especially in the United States. Implementation of environmentally friendly power generation will be a long process and will need to be integrated with the existing infrastructure. A one percent annual increase of green power will gradually diminish our national reliance on fossil fuels, producing political and financial incentives. The diversity of our electric generation sector will reduce our reliance on fuel from foreign markets and diminish U.S. vulnerability to oil price spikes.

As apparent from the Blackout of 2003, a massive overhaul of the current electrical infrastructure is necessary to improve reliability to meet modern digital demands. However, no matter the type of power generation or the amount of money dedicated to renovating the electrical infrastructure, no system could ever guarantee 100% availability. As a result, the job of the Mission Critical Facilities Engineer will persist indefinitely as the sustaining force for 7x24 mission critical operations.

The Blackout of 2003 demonstrated to government officials and corporate society that the backbone of the economy, national security, and the public safety of all citizens depend on reliable power and the infrastructure delivering it. Billions of dollars in business was lost and public safety was compromised as a result of a false sense of security, inadequate backup power protection, and insufficient electrical system maintenance.

As known by experts within the mission critical arena for years and made public by the recent blackout, the current antiquated electrical distribution system cannot meet the demands and reliability needs of today’s 7x24 digital society. Managers of all type and size organizations and institutions need to be well-versed in power risk management and take an aggressive proactive approach to minimize safety hazards and financial damage during a power failure. It is not a question of preparing for if, but rather, for when the next power outage will affect the continuous flow of your business and daily life.

¹ Source: Edison Electric Institute; U.S. Department of Energy
² New York City Comptroller William Thompson
³ Computer Industry Almanac
⁴ Forrester Research
⁵Prepared by a disaster-planning consultant of Contingency Planning Research
⁶ New York City Comptroller William Thompson
⁷ Source: Status of State Electric Industry Restructuring Activity, U.S. Department of Energy
⁸Electric Power Annual 2001, U.S. Department of Energy

Reprinted with thanks from:

http://www.energypulse.net/centers/article/article_display.cfm?a_id=632