Lessons from a Network Outage

Last week a power surge knocked out our Com­cast Busi­ness Class ser­vice for part of 2 busi­ness days. My firm relies on a mix of cloud-based and on-premise IT plat­forms, so this out­age was a reminder that the last-mile infra­struc­ture here is still too frag­ile to shift every­thing to the cloud.

My office is located near a wood­land park with very tall but wind-weakened trees. This means we’re vul­ner­a­ble to out­ages from win­ter storms, as well as mis­takes by repair­men work­ing on the power or cable grid upstream from us. Every year we’re hit by an out­age that lasts at least a day and some­times longer. This year’s out­age reminded me of how much I depend on:

• High­rise to man­age prospect and cus­tomer inter­ac­tions, as well as biz dev planning
• Base­camp for project management
• AT&T to deliver 3G ser­vice to my iPhone so I can retrieve email and tweets when my office net­work is down
• Google and a whole host of Inter­net resources
• Skype for long-distance calls

Unfor­tu­nately, the power surge left my cable modem in a “hung” state, which knocked out DHCP ser­vices behind the fire­wall and thereby broke the con­nec­tion to print­ers and shared dri­ves. The good news is, it only took Comcast’s tech­ni­cian 30 min­utes to get my office back online, once the truck got here. The bad news is, it took 24 hours before I could be in my office long enough for Com­cast to dis­patch a repair­man (which requires a several-hour time window).

Poten­tial Solutions

Elec­tri­cal Outages

Gen­er­ally speak­ing, power out­ages here are more dis­rup­tive than the Com­cast out­ages. This island is cursed with a 40-year-old elec­tri­cal infra­struc­ture that’s vul­ner­a­ble to wind and rain, and is most frag­ile within a square mile of my office. We’ve expe­ri­enced 2 power out­ages that lasted more than a week, and sev­eral more than a day. (It’s hard to believe we’re only 8 miles from down­town Seattle!)

The best solu­tion to power out­ages of this mag­ni­tude is a gas-fired gen­er­a­tor that’s wired to the building’s elec­tri­cal panel — a solu­tion that car­ries a $7000 price tag. One of these days I’ll have to take that step. Espe­cially if global warm­ing con­tin­ues to increase the num­ber and sever­ity of wind­storms off the Pacific.

A less expen­sive approach is a bat­tery backup; how­ever, most of the out­ages here last for hours and some­times days — well beyond the rated duty cycle of most home-office UPC sys­tems. This doesn’t appear to be a prac­ti­ca­ble solution.

Cable Net­work Outages

Work-arounds for Com­cast out­ages are more com­plex, as they depend on whether the cable modem itself is func­tion­ing. When the cable modem is work­ing, even if the cable is dead, my local net­work can still func­tion prop­erly and han­dle local print­ing and file serv­ing, as well as Intranet services.

Thank­fully, my iPhone keeps me cur­rent on emails and Twit­ter, as well as my High­rise CRM sys­tem via an iPhone app. It’s a life­line, but not suf­fi­cient in itself. The prob­lem is, A&T’s net­work capac­ity is not reli­able. Even though I have direct line-of-sight to a cell tower, there’s so much demand for band­width that I often get a “can’t acti­vate the cel­lu­lar data net­work” error message.

When the office LAN is still oper­at­ing I can access key project files, so all I need is an interim alter­na­tive to the cable net­work. One option is to go online with a 4G ser­vice from Clear­wire. But there’s a big down­side: their cur­rent USB modem does not work espe­cially well with Macs, and their 4G net­work cov­er­age is still spotty. The addi­tional monthly cost of $50 on top of Comcast’s $100 fee is a bit daunt­ing for some­thing I wouldn’t use on a daily basis.

Seat­tle is one of the lucky loca­tions to have this 4G option from Clear­wire; how­ever, it down­grades to 3G in most other cities. So far this value propo­si­tion hasn’t been com­pelling for an occasional-use model.

Another alter­na­tive is to leave my office and work in a pub­lic loca­tion with WiFi ser­vice and a per-diem WiFi sub­scrip­tion (such as Boingo’s). This is fine as long as I don’t need to work on clients’ con­fi­den­tial files. For this to be a prac­ti­ca­ble solu­tion for an entire work­day or more, I’d need to invest time and energy in a scale-appropriate strat­egy for back­ing up key files to a cloud-based ser­vice provider. And I’d have to find a secure loca­tion where I could work, use the phone, and not expose my clients’ doc­u­ments or busi­ness issues to pry­ing eyes or ears.

Solv­ing the doc­u­ment access issue is eas­ier than find­ing a secure loca­tion for work­ing on an interim basis. There are a num­ber of web-based stor­age and archiv­ing options, such as Drop­box and Box.net, as well as a vari­ety of prod­ucts that act as front-ends to Amazon’s S3 cloud-based data cen­ter. They’re focused on col­lab­o­ra­tive access to files across project teams, which can intro­duce risk when your work is cov­ered by NDA.  So even to use one of these seem­ingly sim­ple ser­vices requires advance plan­ning (and prob­a­bly an IT consultant’s help) to address access poli­cies and priv­i­leges so as to not jeop­ar­dize clients’ secrets.

This out­age is a reminder that the last-mile infra­struc­ture is still frag­ile, not 100% reli­able 24x7, and there­fore it’s pru­dent to invest in fall-back options even in technology-savvy places like Seattle.

Tagged as: infrastructure for small tech firms