Thursday, September 06, 2007

Stories...and a windy storm

In reading the book "You, Inc" I came across an item that talks about the importance on stories. Not inventions of the mind, but really describing to someone what happened, how you reacted and how you felt. That kind of story. So I am going to start telling more stories.

Yesterday I found classic, but terribly frustrating failure scenario. We had a power outage across the valley and that included one of my clients. The two floors of their building are connected by a decent 24-port linksys switch. Come Wednsday morning half of the computers and phones didn't work. We happen to use Asterisk and VoIP phones. I also happened to have rolled out bad configs the night before. So I regenerate the configs and reboot the phones. After that, in a particular department all the phones still didn't work. It took sometime to find out that the computers that were plugged in next to those same phones also didn't work.

So while half the floor didn't work at all, the other half worked fine. While I was racking my brains to figure it out and running around looking for the problem, people were mentioning minor issues like: the printer was slow, email was slow, the phone quality is poor. At first I just figured that the small issues could wait, I started to wonder. After thinking for a bit that something on the network was overwhelming it, it dawned on me. The switch wasn't on a battery backup. The power storm might have "given it a concussion". After getting the head honchos off the phone so I could power cycle the switch, remember it was only one department on the floor that had the problem, I rebooted the switch and like magic all the problems went away and all the computers and phone worked again.

I made sure to put that switch on a battery backup after that and plan on adding monitoring alarms for out of sync changes to the phone configs.

No comments: