Do you have an "Oh S***!" story?

Monkerz

IT related of course. I am talking about those incidences where your actions did or could have resulted in your termination. A mistake you made, something you said, a cable you tripped over...

I don't have one as of yet, but I'm keeping my fingers crossed it I don't ever have one.

However, I do have a funny, well I think it's funny, story of a prank that nearly landed a co-worker in the government cheese line.

To start this off, my boss has no sense of humor whatsoever.

The "class clown" of our NOC decided it would be a good idea to pull a prank on our boss a few years ago.

The prank started off with two fellow employees randomly hanging fiber and copper patch cables from the clown's body. This happened between our enormous server racks so the boss couldn't see. Once the guy was completely entangled in cables, he was laid on the floor behind our core rack which housed two 6513's and two 6509's. This corner of the DC shares a wall with our manager's office.

Beforehand, we located our manager's port on the 6513 and I remotely consoled into the switch. Once given the signal, a loud bang on the connecting wall, I was to shut down our manager's port. So I did, it took 5 seconds for him to realize his connection was lost and go investigate. His door swung open and he stormed through the office, I followed. The rest of our team was already in the DC, for we could see them through the glass security door. Our manager scanned his card, opened the door, and yelled, "What the f*** just happened!". At that point, the clown screams for help and begins to get up from behind the rack. We all walk around the corner to see him covered in patch cables, with his hands up he states, "I fell off the ladder."

I had never see our boss as mad as he was. I mean I could almost see steam coming from his ears.

The clown let it steep for a few minutes before breaking the news that it was just a prank.

The boss man said, "Get in my f****** office now!". Clown was forced to take a week unpaid off.

Hilarious!

jamesleecoleman

Sounds like the boss was working on something important. I can understand someone not having a sense of humor though.

CodeBlox

HAHA! I actually found it pretty funny! I bet some of the people who can relate to your boss wont agree though. He was probably busy. And no, I don't have a story like such.

rsutton

Gotta be careful with that stuff. If he was downloading a large file or on a conference call it could be very irritating. Personally I don't want to be seen as the joker type at work.

mikedisd2

Yeah, I don't care for the joker types, they tend to be more immature than funny. Some people have to realise that they're not in school anymore.

ziggi138

I have an oh poop story.

While i was working at my last job, i was punching down a few links into a patch panel. the panel was at the bottom of the rack, so i lay down and start punching away. Well apparently i bumped the button on the "pdu" (it was a surge protector that you can buy from walmart.) with my knee and shut down our core routers, core switch, and distribution switches. Now, it took 10 min for the network guys to realize that the network was down, and another 10 min for them to figure out what i had done. I didn't really get griped out, but they did make fun of me for bringing down the network. In my defense, the power strip shouldn't have been on the floor, and they should have used the redundant power setup the way it was intended.

Needless to say, the next day, they were in there cleaning up the rats nest and ordered rack mount PDUs.

Oh, and at my current job (a data center) I shut off the breaker to a customer's cabinet. Thankfully they did have redundant power and there was no down time with them. I was new at the time, so they forgave me and i learned a valuable lesson that day.

bertieb

I love a joke and all, but if someone knowingly unplugged my LAN connection when I was in the middle of something important just 'for a laugh', I'd get them sacked for messing about and then would get myself sacked 'cos of the GBH I'd cause in the aftermath. I've had enough of immature morons disrupting my (valuable) time to get a cheap laugh from other colleagues.

I've had plenty of those moments though, especially in the early days when I used to rack and stack kit. VERY easy to dislodge something, knock a power socket out etc.

Turgon

I will admit to one, although it simply resulted in me having more problems on my hands at work rather than a termination risk, there was never any prospect of that as I was holding everything together..

Bad installations of active directory and hosted exchange servers thrown together by a previous MCSE who was inexperienced and had no concept of good design process or maintenance. They had to let him go. He left no documentation or backups worth a candle for a small but busy hosted datacentre. Not even a network drawing. Exchange server which was also a domain controller with no proper partitioning of the disks and constantly low on disk space. We did some cleaning to restore the service including the removal of applications, one of which was IIS, it's not a webserver right? Wrong. Exchange 2003 had dependencies on IIS. Disk space freed up but a mail server with even more serious problems now. The company did not have any media to do an IIS reinstall, or any backups for that matter. Service was stored with an obscure registry hack provided by a outsider. I got fed up with the place soon after and moved on to pastures new. They ceased trading 18 months later. The whole place was a mess when I arrived, but at least a documented mess when I left, with some improvements when I had time to introduce them which wasn't often because the whole thing kept breaking and we were constantly selling against it. How not to do hosting.

it_consultant

I did something that did result in termination. I deleted 4 weeks of financial data. What made it so stupid was I took a backup and forgot to copy it off of the server, got called away to do something mid-project. I can't describe the pit feeling in my gut when I realized what I had done.

Turgon

it_consultant wrote: »

I did something that did result in termination. I deleted 4 weeks of financial data. What made it so stupid was I took a backup and forgot to copy it off of the server, got called away to do something mid-project. I can't describe the pit feeling in my gut when I realized what I had done.

Sorry to hear that. I think its an example of just how responsible our jobs are and how important it is to leave us alone to do them properly. Many people have jobs that make it difficult to fire them, we are not so lucky.

Aldur

So, on my last job I was working in the lab to test one feature or another. I was sending 1 gb/s of traffic through the routers to test some failover scenarios. At the end of the day I was distracted by something and left without shutting down the traffic generator…

When I came back on Monday I found out that I inadvertently launched a DOS attack

The series of unfortunate events are as follows.

1. In my test bed I was sending traffic to the 7.7.7.7 address, should have used some random RFC 1918 instead.

2. Somebody went behind me and killed the link that this traffic was running over.

3. There were some ALU boxes in the lab that were connected to my routers. At this time they hit a bug that injects a 0/0 default route when it’s not supposed to.

4. The traffic that was getting black holed on my box now had somewhere to go.

5. From the ALU box the traffic made it to an Internet aware router in the lab which then sent it up the line.

Nothing big came of it, but there was some unhappy people. I did learn to be much more careful in my lab testing. Because if something can go wrong, it will.

Jinuyr

My story goes back to 2001 right after high school and my first job working in the hospitality industry. I had just finished my internship at the Corporate Office as a Tech Apprentice when I was promoted to work at the centralized Help Desk for the company. I was trained how to use our AS/400 system where the hotels operated their CMS and LMS system. Within a few weeks, I sent a command to shut down all the terminals in the entire hotel. Not realizing what I had done, I continued working and taking an increased number of support calls.

By the end of the week, we had found that I was the one that sent the command to the system through our audit trails and reports. My boss had asked if I thought that I had done it, I didn't deny it and accepted the possibility of having sent the command. He was satisfied with my confession and willingness to cooperate so I just had a small talk with him about making sure I was checking my work before sending commands to the system. He added that in order for an action like that to have happened, all system wide commands were sent to a computer operator to verify. Apparently the computer operator thought it should have gone through so they sent it anyway >_< He didn't hold me responsible and I was soon promoted to a Corporate Technician a bit later.

Talk about dodging a bullet. On top of it, this was a gambling hotel and it was on a weekend.

undomiel

Sadly my story dates back to just a couple hours ago. Was up all night attempting to P2V a particularly ornery Win2k server. Gave up for the night and cleaned up the mess left in Hyper-V then went to bed, only to be up a bit later finding that I had deleted the wrong server. Now having fun restoring from backups that apparently have not been working since April plus the support contract with their third party software vendor ran out as of today. So this'll be fun.

RobertKaucher

undomiel wrote: »

Sadly my story dates back to just a couple hours ago. Was up all night attempting to P2V a particularly ornery Win2k server. Gave up for the night and cleaned up the mess left in Hyper-V then went to bed, only to be up a bit later finding that I had deleted the wrong server. Now having fun restoring from backups that apparently have not been working since April plus the support contract with their third party software vendor ran out as of today. So this'll be fun.

I have heard this story so many times that whenever I need to delte something or clean up a system like this I
1. Script it so that I do not accidentally click the wrong thing.
2. Always have someone verify my work what I am about to do before I do it.

I have had a few people get annoyed that I am making them check my work, but I really don't care.

undomiel

RobertKaucher wrote: »

I have heard this story so many times that whenever I need to delte something or clean up a system like this I
1. Script it so that I do not accidentally click the wrong thing.
2. Always have someone verify my work what I am about to do before I do it.

I have had a few people get annoyed that I am making them check my work, but I really don't care.

Scripting's a good thought. I'll have to see about how I can put that to work here. Unfortunately number 2 won't really work for me since at my employer there is too wide of an experience gap in most of what I do, so I don't really have anyone to bounce things off of. It would be great to have someone to double check things with though.

it_consultant

Turgon wrote: »

Sorry to hear that. I think its an example of just how responsible our jobs are and how important it is to leave us alone to do them properly. Many people have jobs that make it difficult to fire them, we are not so lucky.

I deserved what was coming, I have been much more careful since and I won't touch a production system until I know it has been backed up and stored on another system.

the_Grinch

I have one that is pretty good (not something I would have gotten fired for). At my last company they used an old Windows 2000 box as a print server for the Finance Department. Now these people were especially bad with technology, so I would get the I can't print call all the time. So I go up, make sure the printer is on, and do some test from the desktops. Nothing. Now, this box had an issue where the print queues would fill up. Due to it's age and some issue, you could not RDP to the box. So I had to walk half a mile one way to get to the server room and get into the box. Get into the box, kill the print spooler, and start it back up. No one in the room, so I walk all the way back. Ask the users to start testing and they still cannot print. So now I am pissed. Shut off the printer, turn it back on, and still nothing. Switch out the cable, still nothing. Reinstall the printer on a machine and still nothing. Finally, I check the wiring closet and everything looks good. Start yelling in my office, etc and my boss says did you check the wiring closet? I say of course and he says, well check one more time. So I look again and don't you know that port was unplugged. We had put battery backups in that weekend and had mounted a switch which required removing and plugging back in some lan cables. In the nest, we missed one

Other story I have is from when I worked for a school district and I thought for sure they would can me. VP at the high school basically brought his PC down because he said it was junk and useless. Admin at the high school says your getting a new one, but you'll need to keep the old one for now. He goes on vacation and the new PC comes in. Admin asks me to install it and says that there shouldn't be anything on it locally everyone saves to the network. I go in, unplug everything, and install the new one. I was working on another project refurbishing old pcs for the students to use in the library. I need three more and his was one of the models. So I wipe it, install a new OS, and put in on the network. The next week I come in and the admin says "where is the VP's PC, he had files on it and I don't know which one here is it." I then tell him that I wiped it and his jaw drops. An hour later I get called to the Director of Technology's office. She in turn talks to me about something completed unrelated (the admin told me he had let her know what had happen). I come back and he asks what she said? I said she didn't ask me about it and he laughs and says we'll leave it at that. An hour after that the VP calls and says that his sound doesn't work. I go to check and see the cable was too short to hook up and I missed it, in which time he starts yelling about how the admin not only lost his stuff, but didn't hook his pc up correctly. At that point I let him know that I had wiped it and boy did I regret saying that lol

Turgon

it_consultant wrote: »

I deserved what was coming, I have been much more careful since and I won't touch a production system until I know it has been backed up and stored on another system.

Very wise.

jibbajabba

I bend over to pick something up and my butt hit the breaker, shutting down the power of half the DC floor. Needless to say it was a *bleep* design to start with (we just rented several suits in there) - but the DC owner weren't impressed to say the least.

Different DC - colleague moved an empty cabinet just an inch back, not knowing that the DC staff, when the racks were put in, wedged a fibre between cabinet and floor tile.

That inch of movement was enough to cut the dark fibre (connection between DCs), effectively cutting off one of our remote sides ..

Whoops...

Lazydog

it_consultant wrote: »

I did something that did result in termination. I deleted 4 weeks of financial data. What made it so stupid was I took a backup and forgot to copy it off of the server, got called away to do something mid-project. I can't describe the pit feeling in my gut when I realized what I had done.

At a company I used to work at they had opened a service center overseas. They thought they would save themselves time by automating things like server builds. One weekend their script took a turn for the worst and started reformatting production hard drives. brought down 153 servers before anyone noticed. Needless to say no one was fired and it took close to a week to get all the servers up and running again.

shodown

This week

Uploaded some files into a UCCX server that shouldn't have caused any problems, but it brought down a almost 300 seat call center for about 90 seconds.


A while back.

A company sent in a ticket to replace a WAAS module. They didn't have any IT guys on site, so it was Cisco 4 hour replacement team that showed up. The company wanted to replace it during business hours since they had redunant routers. Well somebody sometime back thought it would be a good idea to load balance across both routers so when the WAAS module was replaced that entire sites call center went down for about 30 min while the router was down and the WAAS module was coming online.


Working with another company they had a tunnel that wouldn't come up. Its a documented bug in the firewall. So they were going to have to restart there main ASA. This was going to bring down close to 50 sites that had tunnels setup. The site that was down had a big meeting going on and the risk was justified. The HQ router was restarted and the tunnel came up. Other tunnels didn't!!!!. It was a long 20 min (it felt like a hour) bouncing the tunnels that didn't come up.


Another time. A guy got fired, he forwarded the main line number to 911. By the time I got the ticket it had been that way for a 20 min. Imagine a large call center where they get a lot of volume all there users being forwarded to 911.



Had to do a IOS upgrade over the internet with a company who was too cheap to buy larger flash for a device and the router was in some far out warehouse overseas where there were no English speakers. Someone was going to have to fly out there if it went back. No RDP to the 2 machines they had in the factory. Deleted the IOS from Flash, uploaded the new one. Placed the new one in the boot sequence and setup a TFTP server on my computer from home with a IOS in case the boot sequence didn't work. Set the pings to the device and started the reload.


I just sat there. I don't know what the problem was but that device took close to 30 min before pings started to return.

MentholMoose

Probably the worst I've done is delete some files on a Linux file server. In a Bash session over SSH, I meant to type something like, "file *". Unfortunately, I didn't notice I had previously had typed "rm" at the prompt, so when I pressed "Enter" the command was "rm file *"... i.e. very, very bad. I quickly noticed what I had done (there were tons of errors about directories not being deleted) and started frantically pressing CTRL+C, but there was no response, the session appeared to freeze.

I opened a new session to try to "killall rm" and found that the "rm" had mostly failed for some reason, only a few files were deleted, which I restored from backup. I still don't know exactly why it failed, but I'm glad it did!

shodown wrote: »

Another time. A guy got fired, he forwarded the main line number to 911. By the time I got the ticket it had been that way for a 20 min. Imagine a large call center where they get a lot of volume all there users being forwarded to 911.

Wow, was that an accident, or did he want to get fired??

Turgon

Aldur wrote: »

So, on my last job I was working in the lab to test one feature or another. I was sending 1 gb/s of traffic through the routers to test some failover scenarios. At the end of the day I was distracted by something and left without shutting down the traffic generator…

When I came back on Monday I found out that I inadvertently launched a DOS attack

The series of unfortunate events are as follows.

1. In my test bed I was sending traffic to the 7.7.7.7 address, should have used some random RFC 1918 instead.

2. Somebody went behind me and killed the link that this traffic was running over.

3. There were some ALU boxes in the lab that were connected to my routers. At this time they hit a bug that injects a 0/0 default route when it’s not supposed to.

4. The traffic that was getting black holed on my box now had somewhere to go.

5. From the ALU box the traffic made it to an Internet aware router in the lab which then sent it up the line.

Nothing big came of it, but there was some unhappy people. I did learn to be much more careful in my lab testing. Because if something can go wrong, it will.

This is precisely why any test model should be isolated from the production network.

Turgon

Gomjaba wrote: »

I bend over to pick something up and my butt hit the breaker, shutting down the power of half the DC floor. Needless to say it was a *bleep* design to start with (we just rented several suits in there) - but the DC owner weren't impressed to say the least.

Different DC - colleague moved an empty cabinet just an inch back, not knowing that the DC staff, when the racks were put in, wedged a fibre between cabinet and floor tile.

That inch of movement was enough to cut the dark fibre (connection between DCs), effectively cutting off one of our remote sides ..

Whoops...

Always request a site audit for cables before you move racks in a data centre

shodown

MentholMoose wrote: »

Probably the worst I've done is delete some files on a Linux file server. In a Bash session over SSH, I meant to type something like, "file *". Unfortunately, I didn't notice I had previously had typed "rm" at the prompt, so when I pressed "Enter" the command was "rm file *"... i.e. very, very bad. I quickly noticed what I had done (there were tons of errors about directories not being deleted) and started frantically pressing CTRL+C, but there was no response, the session appeared to freeze.

I opened a new session to try to "killall rm" and found that the "rm" had mostly failed for some reason, only a few files were deleted, which I restored from backup. I still don't know exactly why it failed, but I'm glad it did!



Wow, was that an accident, or did he want to get fired??

He was already fired, but had a good knowledge of telephony systems. It would only take him 10 seconds to do it and he didn't have to be anywhere near a computer.

N2IT

it_consultant wrote: »

I deserved what was coming, I have been much more careful since and I won't touch a production system until I know it has been backed up and stored on another system.

I used to know a sys admin who would crush mid range servers all the time. You would tell him not to touch the AS400's and RS6000's while full back ups were runnings and he would kill them anyway. He just loved to crush databases. He was eventually terminated.