Never let a software guy near hardware!

Pilot fish gets an early evening call from his business's security company, telling him that the computer room is sending out a "high thermal event" alarm.

There's no answer after several tries to contact the tech who's on call, so fish makes the 45-minute drive to the office.

"I assumed the office must have lost power and the IT systems were still running on battery backup," says fish.

"While the general office space was fine and showed no evidence of power loss, the computer room felt like Arizona -- the thermometer read over 90 degrees -- and the two redundant wall-mounted air-handling systems weren't running."

Some of the older hard drives are making horrible screeching sounds as fish gets to work. He props open the computer room doors and then begins to shut down all the servers, arrays and network hardware as fast and safely as he can.

Once that's done, he puts some fans in the doorways to draw cooler air in from the main office space. Then fish turns his attention to the pair of air handlers, wondering how two systems running on separate electrical circuits could have died at the same time.

He pushes the power button on one. It roars to life, pumping out cold air.

He flips the switch on the other air handler. It starts right up too.

A few minutes later, the on-call tech arrives with one of the company's software developers -- and an explanation.

"The two had been out golfing at a course about two miles from the office after work," fish says. "The tech put the on-call cell phone, which was set to vibrate, in his golf bag. He never noticed until they were finished with the round that he had voice mails from me."

Then fish describes what he found in the computer room -- the Arizona-like heat, the screaming disk drives, and the air handlers that were switched off.

That's when the developer groans. Turns out he had been in the computer room a few hours earlier, troubleshooting a problem over the phone.

But between the noise from the servers, disk arrays and air handlers, he was having a hard time hearing the person on the other end of the line.

"Figuring it would only be a few moments, he turned both air handlers off to quiet things down," says fish. "But he forgot to turn them back on when he left.

"Fortunately, the damage was minimal -- we lost one drive in a RAID array and missed our nightly backup cycle, but all of our systems were working, communications to the remote offices were back up and our data was safe.

"And I got movie passes and a gift card to a restaurant from the developer as an apology."

Sharky's facing a long, hot summer if he doesn't get some more true tales of IT life. Send me yours at sharky@computerworld.com. You'll score a sharp Shark shirt if I use it. Add your comments below, and read some great old tales in the Sharkives.

Now you can post your own stories of IT ridiculousness at Shark Bait. Join today and vent your IT frustrations to people who've been there, done that.

Copyright © 2010 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
 
Shop Tech Products at Amazon