Power Outage At The Epicenter

Downtown SF is increasingly becoming the epicenter of the tech world (which is why Christian Perry and I picked the name Epicenter::SF for our new events business), and nothing demonstrates this more clearly than the effects of the recent power outage because of an earthquake:

[from Power restored in San Francisco]

[...]

The problem began when breakers in the utility's transmission service opened for an unknown reason, Chiu [spokesperson for PG&E] said. Every time workers attempted to close those breakers to restore service, it caused voltage fluctuations -- high and low flows of electricity through the system -- that impacted PG&E's Martin Substation in Daly City, she said.

That, in turn, caused a transformer to fail in a manhole under 560 Mission St. Witnesses reported hearing a blast shortly before 2 p.m. and then seeing flames licking up through the manhole grate. Chiu said PG&E could not confirm that an explosion had occurred. No one was injured.

However, the incident recalled an August 2005 explosion in an underground vault at Post and Kearny streets that critically burned a woman who was walking by. At the time, PG&E blamed high levels of moisture in oil of high-voltage chambers attached to the underground transformer and said it was checking the safety of about 1,000 other high-voltage chambers.

[...]

For many customers, electricity flicked on and off at least five times before it was finally restored. Several people were briefly stranded in elevators at 201 California St., said security guard Robert Childers.

The data center 365 Main -- which hosts a number of Internet sites, including Craigslist, Yelp and RedEnvelope -- also lost power, causing frustration among many Web surfers.

Several of the sites posted messages to explain to their users that they were having difficulties, while others said they were undergoing scheduled maintenance or upgrades.

After about 45 minutes, power to the data center was restored by back-up generators that remained in service until PG&E could confirm that its supply was stable. There was no immediate response from the facility as to why the generators were not turned on right after the blackout.

Some of the sites, including Craigslist, remained down even after power was restored, as administrators ensured that data in the server hadn't been damaged, among other checks.

Six Apart's data center -- which support Typepad, Vox, and Live Journal -- was out for part of the day. Incidentally, they host my Typepad blog there, so I was hors de combat, as well. Technorati was also down.

The tech world's modern, shiny, gleaming carapace relies on the aging infrastructure of the now-aging information era and the rusting and creaking late industrial era below that. We are only a few tremors -- or a tsunami or hurricane -- away from dropping back into a pre-wired reality.

Can't we do something like the Internet for the power grid? Why can't we have a more distributed, redundant, and fault tolerant system?