Tales from work

Work long enough for any company and you end up collecting a bunch of bizarre and/or stupid stories. Here is one of many...

I came in one hot summer morning to find an ominous wet patch on the carpet outside the server room. I go inside and find the aircon unit is spraying a big torrent of water over everything including the main database server. The main aircon unit had failed because it wasn't powerful enough for the load and had an incorrectly installed drainage line. It had been like this for years as well so it was a mystery why it had suddenly decided to just die in such spectacular fashion whilst still cooling.

Everything was still working even though the underfloor area with all the 3 phase cabling was deep with water and the actual servers themselves were getting drenched. I didn't really want to hang around too much too much in there for obvious reasons. I'm sure management would have preferred me to just go running in to save the systems however

I was the first person from IT in at the time as well. I shouted to get a random member of senior management to come over whilst I tried to remotely shut down all the important servers. After getting management approval aka a Get Out Of Jail Free card I isolated the UPS and generator from the server room so we could investigate without getting electrocuted. Meanwhile, everybody else was reduced to the emergency backup of a pencil, a ledger book and a calculator

The disaster recovery plans were put into play and I had to leave for the cold site to bring everything up whilst other members of IT dealt with the impromptu swimming pool and wet servers. Amazingly enough, everything pretty much survived the ordeal. The main casualty was the comfy wheely chair we had in there to sit on whilst waiting for things to complete. The bottom of one of the AS400s got a tad rusty which always gets weird comments from service engineers. The room had a horrible wet musty smell for months afterwards as well. We insisted that management approved a proposal for an environmental monitoring alarm system which they agreed to with no hesitation

The stupid part about this was that somebody had spotted the wet patch outside the door before I had even turned up. They didn't do anything about it because it was ITs problem not nothing to do with them *sigh* We even got complaints from some of the users because they couldn't use the systems. I guess they expected us to just put a tarp over the servers...

Find more posts tagged with

Free for TechExams community: Cybersecurity salary guide

Compare cert salaries and plan your next career move

Button

Comments

wweboy

Hmm I guess mine would be the very first time I ever got drunk and I mean **** faced drunk I had to come in the next day (a saturday) and rewire the patch panel some how I did good.

Another time I was told to go shovel the snow because the alarms kept going off because snow kept getting between the door and inside and the door wasn't quite shutting (people slammed them until snow was compacted in the door wedges) so because IT took care of the alarms I had to shovel the snow from the door areas.

I guess the time I spent working for my first company I got pretty darn lucky nothing bad happened but I got told of a time when the network admin was just starting and every department took care of their own IT needs.

The data processing department's novell e-mail system was taken over by german spammers. that was 1998 it took until 2007 to fully get them off of netware I was so happy when I got to reformat that server!

Sorry nothing super entertaining me I can't wait to read more!

dynamik

This is the most random post I've seen from Tiersten. Good stuff

My favorite story is more on the software configuration side of things. I've now come across a couple of businesses that have created identical domains (i.e. company.com) and different DCs, yet they are in different forests. Instead of joining the second server to the domain, they created a new forest and joined them with a trust. It was absolutely baffling how they worked with Exchange, users, shares, printers, etc. Total nightmare.

tiersten

dynamik wrote: »

This is the most random post I've seen from Tiersten. Good stuff

Yeah. I was bored as you can tell

The ancient PABX system we used had developed a fault and because it was so old, it was nearly impossible to find parts for. The service engineers must have been trawling eBay to find parts for this thing because none of the parts were new. They weren't able to fix it despite multiple attempts and told us to just buy a new system. The fault was that every so often the entire PABX would crash and stay hung with no incoming or outgoing calls. The "fix" they recommended until we could get a new system was to toggle the power switch. I drew the short straw and had a nice quiet week down in the basement wiring closet with a book...

darkerosxx

Co-worker of mine deleted the entire external DNS zone file for the ISP we work for. Thank you, backups!

djhss68

darkerosxx wrote: »

Co-worker of mine deleted the entire external DNS zone file for the ISP we work for. Thank you, backups!

Wow. I imagine he was terminated without question.

cerberos

darkerosxx wrote: »

Co-worker of mine deleted the entire external DNS zone file for the ISP we work for. Thank you, backups!

loooooooooooool what a stupid!!! Sorry no offense, was it too early to kill somebody? Huh?!

cerberos

djhss68 wrote: »

Wow. I imagine he was terminated without question.

I believe that if he was able to keep his pants on, and wasn't snatched from him for some reason

, that's a big victory for him, no one should ask for a better favor...

arwes

While working for a local ISP (not the one Darkerosxx works for, though I did time there as well..got a story from there too), I had a customer call in about problems with her WebTV not working right. Our official policy was that we give them the local dialup # for their area, the DNS settings and all the mail server settings & such. That was the extent of our support. Anything beyond that, we told them to call 1-800 GO WEB TV. She had all the correct settings, and it just wasn't working so I sent her on her way like we always did.

She called back about 5 minutes later and just happened to get me. She was literally screaming at me, saying she's going to turn us in to the BBB and how dare I send her to something like that. At this point I'm thoroughly confused. I called 1-800 GO WEB TV and immediately got through to their customer service dept. Then it hit me. Most of our customers are idiots. How would an idiot dial that number? A-ha! They'd replace the 'O' in 'GO' with a ZERO. I called it, and yep I got a message stating that I've "reached the hottest chat line in America, 1-800 (slang for female anatomy)". I tried explaining to her what she'd done and she still demanded to speak to my supervisor. Knowing that place, they probably gave her a free month of service just to keep her.

Pash

arwes wrote: »

While working for a local ISP (not the one Darkerosxx works for, though I did time there as well..got a story from there too), I had a customer call in about problems with her WebTV not working right. Our official policy was that we give them the local dialup # for their area, the DNS settings and all the mail server settings & such. That was the extent of our support. Anything beyond that, we told them to call 1-800 GO WEB TV. She had all the correct settings, and it just wasn't working so I sent her on her way like we always did.

She called back about 5 minutes later and just happened to get me. She was literally screaming at me, saying she's going to turn us in to the BBB and how dare I send her to something like that. At this point I'm thoroughly confused. I called 1-800 GO WEB TV and immediately got through to their customer service dept. Then it hit me. Most of our customers are idiots. How would an idiot dial that number? A-ha! They're replace the 'O' in 'GO' with a ZERO. I called it, and yep I got a message stating that I've "reached the hottest chat line in America, 1-800 (slang for female anatomy)". I tried explaining to her what she'd done and she still demanded to speak to my supervisor. Knowing that place, they probably gave her a free month of service just to keep her.

Haha brilliant. You should of asked her what it was like to wind her up further!

arwes

God we had this one girl there that was clueless. People with problems connecting at 56k would call in and we had a favorite site for getting init strings. I told the girl the site to give him (can't remember it at all), but she forgot the 'S' at the end of it. When she hung up, I was worried. I said what site did you send him to? I tried going there and it was a black page with a link that said 'If you are over 18, click here'.

As luck would have it, he called back and got me. He asked if that's a site we go to on a regular basis. I said, not the one that was given to you, no. He said that he's over 18, and wants to get his modem working properly so he clicked. I think he was in shock over the granny pr0n that raped his eyeballs. Yeah, that girl didn't last long.

arwes

OK last one, it's a two fer. In my short stint at Darkerosxx's company, I had to sit through other people's support sessions until I was ready to handle them on my own. The previous company had maybe 2,000 customers all in the south. This company has several thousand spread around the country (farthest away being Gig Harbor, WA...some chick from there was drooling over my accent, while Quality Assurance was listening in LOL). My very first call on my own was from a customer in Ohio, and she was having problem with disconnects. I was starting the troubleshooting process, but she said she already knew why she was being disconnected. See, she got a mailer advertising the faster (but more expensive) DSL service in her area and that we were constantly disconnecting her so that she would move to the more expensive DSL service. I made the mistake of laughing. She of course said, "what's so funny?". I said OH, you were serious? Ended up changing her init string and all was well.

The DSL service we sold was pretty crappy compared to most other ISPs, but we were the CLEC in their areas and that's pretty much all they could get unless someone's offering cable internet there. I can't remember the exact speeds but I think it was 384k down and 128k up. At the time, I had Earthlink cable and my speed was 4Mbit down and 512k up. I had just walked a customer through setting up the PPPoE dialer stuff and some registry tweaks, and during the course of rebooting his PC he asked if I used that same DSL at home and I said no, I use cable. When we got it rebooted, he went to DSL Reports and ran a speed test. It said something like your connection is 8 times faster than a 56k modem! He says, "HA! I bet that beats the **** out of your cable, don't it". Err, actually sir when I run it at home mine says it's 43 times faster than a 56k modem. He says, oh...my DSL sucks. Yeah it does, but it's still better than dialup. In retrospect, I probably should have said "that it does, sir!" but I'm just lucky it was late at night and nobody from QA was listening in.

RouteThisWay

Just had one the other day...

I was providing tech support to our Technical Product Manager's (these people manage the different web products this facility provides). So I basically sat in incase something happened (bad practice my the person I replaced caused this implementation).

They were discussing Drupal. One lady smugly said "so, is it true Drupal is open source?". One of the local experts was giving a presentation on it and replied "Yep".

She got this grin on her face like she just caught someone with their pants down. She had this look like she was trying to bull her way to the top, like she knew more than everyone. She matter-of-factly replied, "That's not true, it is not open source. You have to enter a password."

The room went very quiet and I cried because this technical expert who was managing all of this is making 4x my salary.

arwes

LOL she would be an excellent writer in Hollywood. Did she freelance for the old Nash Bridges show? Video is work safe, I remembered being doubled over laughing about the T-3 line so I hunted down the episode and put the clip on Youtube. The MS Paint as FTP thing I never noticed until I was trimming it down in VirtualDub.

Aldur

RouteThisWay wrote: »

She matter-of-factly replied, "That's not true, it is not open source. You have to enter a password."

hahaha, oh wow, just wow.

RouteThisWay

arwes wrote: »

LOL she would be an excellent writer in Hollywood. Did she freelance for the old Nash Bridges show? Video is work safe, I remembered being doubled over laughing about the T-3 line so I hunted down the episode and put the clip on Youtube. The MS Paint as FTP thing I never noticed until I was trimming it down in VirtualDub.

Favorited that video and sent it out. That is awesome.

You ever seen that movie Breach with Chris Cooper?

Breach (2007)

One of my favorite lines, when he is overhauling the security systems "I want IP routers in EVERY ROOM!"

That show NCIS has some pretty funny IT buzzwords thrown around too. One suspect was behind a firewall, so they "triangulated his position through another router".

cerberos

RouteThisWay wrote: »

Just had one the other day...

They were discussing Drupal. One lady smugly said "so, is it true Drupal is open source?". One of the local experts was giving a presentation on it and replied "Yep".

She got this grin on her face like she just caught someone with their pants down. She had this look like she was trying to bull her way to the top, like she knew more than everyone. She matter-of-factly replied, "That's not true, it is not open source. You have to enter a password."

The room went very quiet and I cried because this technical expert who was managing all of this is making 4x my salary.

Wahhhhhhhhhhhhhhhhhahahahaha, this is more stupid and funnier... This is the kind of things that would make you pee laughing GGGGGGGGGSsssssssssss

Forsaken_GA

RouteThisWay wrote: »

Favorited that video and sent it out. That is awesome.

You ever seen that movie Breach with Chris Cooper?

Breach (2007)

One of my favorite lines, when he is overhauling the security systems "I want IP routers in EVERY ROOM!"

That show NCIS has some pretty funny IT buzzwords thrown around too. One suspect was behind a firewall, so they "triangulated his position through another router".

Myself and some friends turned 24 into a drinking game. Everytime they improperly used technological terms, you had to take a drink.

When Chloe joined the cast, there were some episodes where we were very drunk indeed.