I’ve been a VERY BAD BOY lately about posting to FOIB, which goes to prove that my current man-of-leisure philosophy is going to my head. I wrote this article a few weeks ago, ostensibly for the Vancouver Sun. They of course had their heads inserted in various orifices and due to labour union problems couldn’t publish the piece. By the time they told me the story was cold and nobody else was interested either, so you my friends will read this exclusively unless I can retool it for a California audience.
I’m proud of this puppy.
-Ian.
—-
Strategies for Riding the Electrical Rollercoaster By Ian Andrew Bell, scoop [at] ianbell [dot] com
The growing power crisis in California presents a stark but very real vision of the future for all of us in North America; and as Vancouverites and Quebecers learned this winter, even with our ample supply of hydroelectric power, you just can¹t take anything for granted anymore. As it happened, thousands of businesses and residences were impacted last month by an explosion at a BC Hydro substation, which cut off power to Vancouver¹s downtown core for several hours.
For many businesses, the power outage in Vancouver was a wakeup call  ³This lasted longer than any outage since we¹ve been in business,² said Simon Kan of Dolphin Computers, a computer systems consultant who supports dozens of downtown high-tech companies. ³Our clients¹ battery-backup systems rode out the initial dip in power, but they weren¹t prepared to handle such a lengthy interruption.²
Reports are still coming in, but Simon expects to spend the next few weeks working double-time to assist clients who have suffered everything from software glitches and lost data to catastrophic hardware failures as a result of the sharp cutoff in power.
This is no surprise  power failures are the most common cause of system failure. In fact, damage to equipment, as well as loss of productivity due to power spikes, brownouts, and blackouts cause more than $300 million in PC damage annually.
And that¹s not the only problem. With the Internet becoming a predominant tool for doing business globally, companies who go online significantly increase their vulnerability to damages and lost productivity as a result of the loss of power. According to a study commissioned by Contingency Planning & Management Magazine, companies in the brokerage business and other financial sectors can lose as much as $6.45 million USD per hour of downtime.
Companies who host their own Web Sites, Electronic Mail, and/or File Transfer services directly from their offices go effectively ³off the air² whenever power is lost to the telecom equipment, routers, or servers that carry their information online. The result is incoming mail ³bouncing² back to the senders, remote office workers cut off from their peers, and rather embarrassing ³ERROR² messages confronting customers and partners who are trying to view the company Web Site.
³It’s interesting that it takes a major blackout, or rolling blackouts like those in California, for people to realize the value of a good power strategy,² says Greg Fournier, Sr. Product Manager for American Power Corporation (APC), makers of UPS battery-backup systems and software. ³They can have the most incredible gear and technology, but without power that equipment goes to waste.²
So how do we prevent hardware damage, data loss, confused employees and irate customers? Addressing this threat to corporate integrity requires a somewhat complex matrix of solutions. How companies address these issues depends upon size, budget, and the degree of criticality of their information systems. A strategy that incorporates the flexibility to deal with the changing climate of your business is key  and dependence upon a single solution or technology is a recipe for disaster.
The following three basic categories of services and products combine to ensure a well-rounded and cost-effective overall strategy for corporate survivability during a power failure:
Protecting Your Users
Nothing is more frustrating when you¹re working on that seven-layer Excel spreadsheet than having your computer¹s monitor go black as you¹re entering that masterstroke calculating field. However, most office workstations only require enough power during a failure to ride out isolated brown-outs or brief interruptions. During a power outage, individual users simply require enough time to safely store their data, close their programs, and shut their machine down properly.
APC sells a simple power-bar sized device designed for individual computers. The product, called the ³Back-UPS Office² offers surge protection as well as up to 40 minutes of battery-backed running time for a single computer for less than $120. The software that ships with this device will inform the computer it¹s attached to when its own power is running low so that it can shut itself down safely when the user is not available.
Another option to consider in protecting data on workstations is equipping employees with notebook computers instead of desktops. Because they¹re portable, these come ready with their own built-in batteries, have more rugged internal power circuitry, and have numerous other advantages (and disadvantages, depending upon your office environment). Simply plugging a notebook into the AC circuit via a surge suppressor probably gives adequate protection from all but the most catastrophic of power events.
Keeping Internet Services Alive
Long gone are the days when companies can entrust their corporate Internet strategy to the neighbour¹s prodigious (a polite term for ³nerdy²) teenager. Just as gone are the days when it was a Good Idea to run Internet services from within the offices of the companies they are required to serve. In this day and age, people (and customers) have far greater expectations for the reliability of Internet services such as email and web sites.
Simon Kan sees the value in taking those customer-facing aspects of the business, such as email and web services, and pushing them outside the office: ³There are just too many calamities that can happen to servers in an office environmentŠwhile some services have always got to be in the office, internet services need to be in a place where they are safest to make sure they¹re always up.²
Fortunately, many new Internet businesses have applied a longstanding model for reliability taken from the pages of telecom 101, creating co-location centres with redundant internet connections, battery-backup power, and their own power generation facilities.
One such company, which rode out Vancouver¹s power outage with uninterrupted bliss, is Peer1. Peer1, headed by Geoff Hampson, provides co-location facilities for internet servers: ³We were impressed by how our systems handled this real world test  there is a real difference between scheduling a power shut down for testing purposes and actually losing utility power for several hours late at night. Our customers who live and work outside of the Lower Mainland never even knew there was a problem.²
For a few hundred dollars per month, customers can park their Internet servers in highly secure, temperature-controlled racks in Peer1¹s co-location network centre. Here, the servers will enjoy the utmost in quality in utility power, air conditioning and cooling, disaster protection, and even protection from curious onlookers.
But housing your internet servers in a co-location facility like Peer1 is only an option for companies who have the resources to manage them. For smaller companies without the desire for their own dedicated Internet servers with all of their accordant maintenance costs, outsourcing may be the answer. Fortunately, there is a font of new Service Providers who will host your company¹s Web, email, and other Internet services on their own systems in operations centres similar to Peer1¹s.
Vancouver¹s Electric Mail Company is an IT outsourcing company that provides, among other services, hosted email and messaging services. EMC¹s email solution allows corporations both large and small to cost-effectively outsource their messaging needs while preserving corporate integrity, manageability, security, image, and branding. ³Email is like electricityŠwhen it works, nobody says Œthanks¹, but when it¹s down there¹s an uproar² says Iain Black, EMC¹s President and CEO.
His job may be thankless, but EMC saves companies money while significantly increasing the reliability of their email services. Iain estimates that small-to-medium-sized companies who operate their own internal email servers spend approximately $80-$120 per month per user to keep these services up and running  and that¹s to say nothing of the costs for building a strategy to effectively operate these services through power failures and other disasters  which most haven¹t done.
³Companies should take the time to analyze their critical business applications and decide which ones can be effectively outsourced,² Iain says, ³and those applications should be placed in the hands of companies with both the infrastructure and the skillset to ensure quality and reliability.²
Protecting the Crown Jewels
Says Greg Fournier at APC: “One of the most dangerous risks to any business is the loss of power. Any outages can cause companies to lose important and critical data, and will cause companies lots of problems with failing hardware and downtime.² According to Greg, these costs grow exponentially with the number of seats (employees) affected by the outage.
The servers that store all of your Local Area Network¹s backup files, control the printers, and keep track of users and security are particularly sensitive to outages. It can take days to resurrect one of these monsters when they suffer damage from a surge or a power failure. Worse than the loss of data and/or hardware is the productivity loss that can result from such an interruption and the time it takes to restore services to their former glory.
Whatever the size of your business, you will want these servers to both be able to ride out brownouts or dips without interruption, and for the servers themselves to know when power is lost so that they can shut themselves down effectively and safely.
APC and others make high-end battery-backup units for servers and network operations centers, like APC¹s ³Smart-UPS Rack-Mount². These devices not only add the ability to extend the server¹s operations through a major power failure, but also interact with the server via software, informing it to shut down safely when battery life runs low.
APC¹s software, called ³PowerChute² will even inform Systems Administrators via email or pager when a power event is detected. Properly configured, PowerChute will be able to manage the UPS systems on its own or in conjunction with Enterprise-class network management tools like Tivoli or HP Openview.
Anticipating your power needs in a growing network environment is often where businesses, even those who invest in UPS (Uninterruptible Power Supply) devices, fall over.
Many SysAdmins do not take the time to accurately calculate the power demands (measured in WATTS) of their various pieces of network hardware. As the operations team continuously add new devices, thereby increasing the accordant power draw, your UPS which once gave you 8 hours of uninterrupted power may now provide only 30 minutes. Even worse, some devices which are critical to the network¹s survivability may be plugged directly into AC while nonessential items are occupying valuable power cycles on the UPS.
Implementing a strategy for the salvation of network equipment during a major power outage is not so simple as ordering a RackMountable UPS from your local reseller  it requires planning and careful calculation.
Many companies find, once they do these calculations, that they simply do not have the square footage to house enough battery storage to power their equipment through major interruptions. That¹s where more brute-force methods of power backup come in.
Gordon Roberts, Branch Manager for Finning Power Systems Canada in Vancouver, has had his phone ringing off the hook since Tuesday Night. Finning sells standby diesel generators that output KiloWatts and MegaWatts of power for factories and buildings.
³It used to be that standby generators were an afterthought  customers bought them because they were mandated by the building code, for operating elevators and emergency lighting, and so they did it as cheaply as possible.²
But now, Gordon says, that has all changed. ³The Information Age has created a need for businesses to have absolute reliability of the utility. If America OnLine has a major central outage,² says Gordon, ³it can cost them one million dollars per minute.²
That¹s US Dollars, too. Truly, for companies in the Information Economy, the cost of standby diesel generators, which can run between $20,000.00 and $3,000,000.00 depending upon the required power output, is miniscule compared to the opportunity cost of being ³down², even for a few hours.
Many of Finning¹s customers are building managers whose tenants have banded together to demand higher output from the building¹s existing backup power generators for their data equipment. Whereas one company does not have the need for a 65-Kilowatt generator (enough to power about 130 desktop computers), several can spread the cost of the generator easily  and for creative building managers, selling backup power is an additional revenue opportunity.
The key, according to Gordon, is to build not for today, but for tomorrow. ³Build it and they will come.² If recent events in California are any example, he may be right.
-Ian/2001
Ian Andrew Bell is a technology consultant and writer living in Hollywood, CA.
Links:
American Power Corporation: http://www.apc.com Peer1 Networks: http://www.peer1.com Finning: http://www.finning.com Electric Mail Company: http://www.electricmail.com Dolphin Computers: http://www.dolcom.com