"Oh God Why" Moments~

Mishra

Moki99 wrote: »

Anyone get fired or almost fired doing some of this stuff? I'd assume some of these are no joke when it happened.

Nope. Too valuable and most understand you can't prevent all mistakes.

badrottie

Moki99 wrote: »

Anyone get fired or almost fired doing some of this stuff? I'd assume some of these are no joke when it happened.

Only someone who has worked in an operational aspect knows that due care can only go so far, there is a cone of uncertainty, and human beings are highly-error prone at the best of times.

Compare information technology to medicine, where the main difference is monetary loss to loss of life. We have it easy, in comparison.

Everyone

badrottie wrote: »

Compare information technology to medicine, where the main difference is monetary loss to loss of life. We have it easy, in comparison.

If you work in Healthcare IT it can be BOTH!

dead_p00l

Forsaken_GA wrote: »

That would be the one. dead_p00l, if you're running MX's, make sure JunOS is updated hehe

I know we hit the CPU utilization bug with the last one we rolled out. I'm pretty sure we updated JunOS across the board when we found that but ill definitely double check.

undomiel

badrottie wrote: »

Only someone who has worked in an operational aspect knows that due care can only go so far, there is a cone of uncertainty, and human beings are highly-error prone at the best of times.

Especially when those human beings are going on 48 hours without sleep!

pizzaboy

undomiel wrote: »

Especially when those human beings are going on 48 hours without sleep!

Like some doctors (not that I'm comparing)

joshmadakor

xenodamus wrote: »

I was fumbling around in SCCM a couple of weeks ago, trying to figure out how to re-advertise a MS Office installation to a particular PC.

I accidentally advertised the software to the "All Systems" container, which includes about 2500 desktops and 500 servers. O.o


Luckily I noticed it, and immediately figured out how to delete an advertisement as well. I think my permissions may have been restricted on that box if the install had gone through, lol.

I've seen something similar to this happen...

I've seen a lot of really weird things happen with SCCM and advertisements actually...

CodeBlox

Slowhand wrote: »

I saw this happen once, at a school. A student enrolled in a UNIX class took down a one-of-a-kind server that had been built many, many years ago from scratch by a development team from UC Berkeley. There was no way to recover it, there was no install media of any kind, so they had to scrap it and set up some FreeBSD boxes for the next semester.

The incident happened when the regular user accounts had been disabled and the admin for the lab had stepped away from his chair for a few minutes while looking for some documentation. The student in question came in, saw a console window open, and thought he'd use his "expert" skills to clean up the user directory; the admin was, of course, logged in as root. I'm not sure what happened to the student, I'm fairly sure some disciplinary action was taken against him for wrecking the server and for using someone else's login.

Er... I didn't actually think it would work and tried it this morning on a dual boot setup. Well, right after I did it, my screen went blank and I rebooted. I immediately get an error about some found not being ound and there was a GRUB recover prompt. None of the commands worked at all, not even help. I had XP installed along side this. I pull out my XP disc to do a fixmbr and I can't even remember the damn administrator password. I don't think I even set one... About an hour later, I ended up having to reinstall XP and it's just finished... So if anyone is wondering, rm -r * does really work! Man, I didn't have anything too important but now I have to gather all apps up again...

Monkerz

Can't say that I have an oops moment as of yet, but...

The April fools joke got some people in trouble. We decided to send an email out to all users in our building (around 50ish, mostly IT and accounts payable people). The email simply stated that we were going to be blowing out our voice lines over the weekend and if you didn't want dust, dirt or moisture on your phone and desk; you will need to go the the computer lab and pick up a plastic bag to cover you receiver with.

Needless to say, one lady got pissed off at the inconvenience of having to walk 50 foot to pick a bag and cover her phone. Decided to email the CIO with a formal complaint.

Excellent1

Monkerz wrote: »

Can't say that I have an oops moment as of yet, but...

The April fools joke got some people in trouble. We decided to send an email out to all users in our building (around 50ish, mostly IT and accounts payable people). The email simply stated that we were going to be blowing out our voice lines over the weekend and if you didn't want dust, dirt or moisture on your phone and desk; you will need to go the the computer lab and pick up a plastic bag to cover you receiver with.

Needless to say, one lady got pissed off at the inconvenience of having to walk 50 foot to pick a bag and cover her phone. Decided to email the CIO with a formal complaint.

I actually laughed out loud on that one. Thanks for sharing!

nycid

Rebooting the PDC while I thought I was loggin off.. Remote site over 100 miles away...

Removing a GPO and started deleting AD.... Luckily I was able to stop replication before it went out through the entire network. Was able to recover from BDC

Removing snapshots with the help of VEAM support when we first started using it and hearing the support guy saying "now you didn't just delete what I told you to did you?" Yeah, production server went down and we restored from backup....

End user swapping out tape decided to remove the HDD instead took 100+ user site down until we could get the drive re-installed....

I'm sure there are more... These stick out the most!

Retards,

Slowhand

CodeBlox wrote: »

Er... I didn't actually think it would work and tried it this morning on a dual boot setup. Well, right after I did it, my screen went blank and I rebooted. I immediately get an error about some found not being ound and there was a GRUB recover prompt. None of the commands worked at all, not even help. I had XP installed along side this. I pull out my XP disc to do a fixmbr and I can't even remember the damn administrator password. I don't think I even set one... About an hour later, I ended up having to reinstall XP and it's just finished... So if anyone is wondering, rm -r * does really work! Man, I didn't have anything too important but now I have to gather all apps up again...

And now you've learned the origin of my favorite response to the question, "What happens when I do <blank>?"

It is, and will always be, "Try it. It'll be funny."

cisco_trooper

Monkerz wrote: »

Can't say that I have an oops moment as of yet, but...

The April fools joke got some people in trouble. We decided to send an email out to all users in our building (around 50ish, mostly IT and accounts payable people). The email simply stated that we were going to be blowing out our voice lines over the weekend and if you didn't want dust, dirt or moisture on your phone and desk; you will need to go the the computer lab and pick up a plastic bag to cover you receiver with.

Needless to say, one lady got pissed off at the inconvenience of having to walk 50 foot to pick a bag and cover her phone. Decided to email the CIO with a formal complaint.

I implemented the UpsideDownTernet on our IT department one April Fools day. We had one guy on staff who would surf facebook all day checking out the pictures of hot chics. Well, every single one of those pictures was turned upside down and it took him a good four hours to say WTF. All my pictures are all upside down. LOL. Can't say I'm not glad that guy is gone.

undomiel

nycid wrote: »

End user swapping out tape decided to remove the HDD instead took 100+ user site down until we could get the drive re-installed....

Reminds of something that was told to me by one of the execs of a local company. A different exec went into the server room and saw that there was a flashing light on one of their drives. They popped it out and put it back in and then just to be safe popped out every other one and put them back in. The rest of the day was spent restoring from backups.

UnixGuy

I mispelled "shifting" once, wrote it without the "f"

Essendon

In my previous role, I used to go on a tape run during which I'd swap used media tapes for new ones. Before we left on the run we'd tell the team where we were going. I usually wrote "Heading off for the Eastern Tape Run", but on that eventful day I wrote "Heading off for the Eastern R@pe Run".

Since then I have put a delay of a minute on my emails. Any emails I send out sit in the Outbox for a minute, giving me an opportunity to edit if needed.

chmod

I was working with the core equipments of a mobile company, i was making some changes to the routing from the core to the applications and i was so tired that i did not realized that i mistyped one of the signaling points so the traffic was routed to another box and that box did not know how to route( it back(to at least get an error message or info in the logs). I had set a route for each prefix to the application box(around 20 prefixes), i randomly chose 3 prefixes to test and they worked just fine so i left and went to sleep then at 4am everybody was calling me that one of the services was not working, i did test and it was working just fine after having NOC guys to gather some basic info of the users affected i realized one of the prefixes was wrong and it affected around 20k users.
I expended 1.1/2 hours preparing an explanation about the partial outage.

kurosaki00

Once I miss clicked the stop all button in our Port Tunnel, and pretty much stopped all the connections that were passing through it lol
But instantly like a second passed I just noticed and clicked it again.
Everything ran smoothly lol
thankfully lol

pizzaboy

undomiel wrote: »

Reminds of something that was told to me by one of the execs of a local company. A different exec went into the server room and saw that there was a flashing light on one of their drives. They popped it out and put it back in and then just to be safe popped out every other one and put them back in. The rest of the day was spent restoring from backups.

That was hilarious thanks.