Had my first real "OH ****" moment with a huge mistake I made last night

Well, they say it's bound to happen to everyone in IT (They will make a small handful of really bad mistakes). Well, I never thought my first would happen so soon... I have been doing Systems Administration for 7 months now and it all began last night. I was preparing some workstations to patch last night and so I VPNd in and scheduled the jobs in Kaseya for "All workstations"... I leave OWA open in another tab while I watch some boxing on youtube (Tyson vs Lennox). Suddenly I hear constant chimes for new email in the OWA tab so I check and see tons of errors from our monitoring system. Stuff about the Exchange DAGs having connectivity issues, replication failing, servers going offline. I VPNd back in and saw a fellow coworker already investigating so I pull up Lync and type to him "Something is happening..." and he says "Yeah, a ton of servers just rebooted". At that moment, I'm thinking to myself "OH ****" as I frantically pull up Kaseya to see if what I knew I had done was really done... Sure enough, instead of scheduling "All workstations" to reboot, I somehow selected the "No View" which is EVERYTHING, workstations and servers! Long story short, I fess up and explain what happened at that it was an honest mistake. thank goodness it was an after hours one. Had to get several folks online to verify things were back up and working. I was in a huge panic over this. I was told that every one will make a few big mistakes like this, it's just a matter of not making the same mistake twice. I did get a talking to by the VP of technology about it (I was summoned to come up and talk to him). He understands that it was a mistake and says I'm doing a good job. He did put his foot down however and said not to do it again. Anyone ever done anything like this before?

Find more posts tagged with

SAVE $250 on Your Boot Camp — Use Code "EXAM26"

Exclusively for TechExams members for Infosec Boot Camps starting before April 30, 2026

View Boot Camps

Comments

About7Narwhal

Oh man, if that happened where I work (Medical Provider) I would most certainly lose my job. The amount of data that would be lost, even on systems that back up hourly, would be catastrophic. Not to mention the fact that the DBAs would be out for blood... apparently rebuilding is very time consuming.

On the bright side, your VP said you were doing well!

**EDIT**

So.. how about that stress you had while walking to the VP office the next day? Longest walk of your life. lol

cruwl

I did something like that at my last one, We used a scheduled GPO to patch on friday nights once a month, as days before patching we would have to enable the GPO, once patching was over we would disable the GPO. Well one time I forgot to disable the GPO so the following friday night at like 8pm all the servers patched and reboot out side a maint window. Mistakes happen, take responsibility for, learn from it.

QHalo

When I used to push applications, patches, etc to all workstations, I made sure that I was completely focused while doing it. Triple check to make sure that what I'm about to do won't affect anything else. This stuff happens. I've done far worse and kept my job. At least it was after hours. It could have been during the day when everyone was around. These are the times when you learn to always be careful when even doing the rudimentary tasks. And also, we never had patching of servers in the same interface as desktops. Always completely separate systems.

sieff

I've done worse on several occassions. First IT job with major Service Provider. Dropped an ATM switch node. Probably 1000s of clients passed traffic through that node. Once I did a phone install for a huge hospital and had call routing screwed up. No inbound calls for a few hours... Things happen, you can only grow from it and get better.

alan2308

Yes, and in Kaseya no less. In window 2, I switched to "no view" and promptly forgot. Then in window 1 I hit "select all" which was then everything, not just the 5 or 6 machines in the test machine view still on the screen in that window. I hit "backup now" and left for lunch. My cell phone started blowing up as I was placing my order at Subway.

If you ask me, Kaseya almost encourages you to make that sort of oops once in a while...

QHalo

Kaseya sounds like a piece...

hoktauri

Other than accidentally frying a hard drive (still not sure exactly how) back around high school when I didn't know what I was doing not a whole lot. Screwed up things on my personal PC more times than I can count though, but that's how you learn.

volume

There's a quote by Herbert Hoover about engineers can applies to us IT folk too:

"The great liability of the engineer compared to men of other professions is that his works are out in the open where all can see them. His acts, step by step, are in hard substance. He cannot bury his mistakes in the grave like the doctors. He cannot argue them into thin air or blame the judge like the lawyers. He cannot, like the architects, cover his failures with trees and vines. He cannot, like the politicians, screen his shortcomings by blaming his opponents and hope the people will forget. The engineer simply cannot deny he did it. If his works do not work, he is damned...

On the other hand, unlike the doctor his is not a life among the weak. Unlike the soldier, destruction is not his purpose. Unlike the lawyer, quarrels are not his daily bread. To the engineer falls the job of clothing the bare bones of science with life, comfort, and hope. No doubt as years go by the people forget which engineer did it, even if they ever knew. Or some politician puts hs name on it. Or they credit it to some promoter who used other people's money . . . But the engineer himself looks back at the the unending stream of goodness which flows from his successes with satisfactions that few professions may know. And the verdict of his fellow professionals is all the accolade he wants."

Mrock4

It happens. I still make mistakes often, but I've just gotten better at identifying situations where they are likely to happen and mitigate the risk if at all possible.

And my golden rule is to always be honest when you make mistakes- they actually build your credibility with your coworkers, so that when something breaks in the future, and they ask if you did it..and you say "no" - they'll believe you. So it sounds like you did good.

lsud00d

I still haven't done anything widespread but I thought I made an oops lately when after installing a mainframe batch agent it kept crashing the server nearly instantly...after a few reboots and watching what happened, a coworker realized that since it was recently rebuilt (news to me) a TON of jobs had been queued up and were instantly swamping the agent which caused CPU to spiral out of control and freeze the server...we canceled all the jobs and all was well

jibbajabba

I don't think I'd get away with a verbal "Don't do that again"... Working in finance means server need to run 24/7 and each reboot, unless planned, costs a LOT of money. Especially when for example both nodes in a cluster reboot at the same time etc.

Worse still, some software doesn't work with certain patch levels so they musn't be updated (there are usually things in place that it isn't even possible, but still), so updating certain server means certain reinstalls ...

So yea - good thing it went ok regardless

I think the worst bit is kicking people out of bed because you screwed up haha - but as you say - it happened to all of us. Wait until you accidentally delete the wrong VM from vSphere - back in the day the vSphere client didn't even display the name of the VM in the dialog you were about to delete .. so things ... happened

Also, every time I update an Exchange server I remove myself from the Enterprise Admin group so I don't accidentally update a servicepack (which effectively updates the AD schema) - you learn from your mistakes

Bundiman

You story reminds me of our intern who was re-imaging workstations. He meant to image a dozen but somehow clicked the wrong box and told every workstation in the enterprise to re-image. We started getting calls that workstations had rebooted and even our own were telling the users to wait while the image was applied. He was lucky that there was a 5 min delay when image more then 5 workstations and I had to run and power down the Altiris server. No damage was done and we had a laugh about it later. Today he is the enterprise engineer for the that system

paul78

jibbajabba wrote: »

Working in finance means server need to run 24/7 and each reboot, unless planned, costs a LOT of money.

Same here... And outages on infrastructure providing security controls are a huge deal.

@op - you are lucky to still have your job - and so is your manager. That opps would have resulted in a few terminations at some companies.

FloOz

At least you were able to recover form it. I have yet to have a moment like this...I dread the day it will happen though

gorebrush

Rebooting all servers isn't honestly the worst thing you can do.

My best one is an Exchange 2003 upgrade project that went really badly wrong. Well...

We had two servers. One main Exchange server, that ran Windows 2000 and Exchange 2000. I upgraded, in place, to Exchange 2003, and then upgraded Windows 2003, again in place. This was on a HP Server, DL360 G4 I think it was.

The upgrade went absolutely fine, with no problems whatsoever. That was Saturday's job done.

Sunday, I was to follow the same procedure to upgrade our BACKUP Exchange server, which was our PRODUCTION SQL Server...

I upgraded Exchange first, again, in place, with no problems. Then came the real fun. I popped the Windows Server 2003 disk in and went ahead with upgrading. This was a DL360 G3. This is where the wheels fell off, the server ended up in a reboot loop.

After a panic, I luckily was able to get hold of a friend of mine who knew quite a bit more about servers than me, and then taught me about the HP SmartStart CD... WOOPS!

I started afresh and managed to get the server back online. However, our production SQL Server was goosed, had to reinstall that and restore from backup. We had to just recreate usernames for our 100 employees on the Monday morning. Waste of a few hours, which knocked the company back a bit that morning, but apparently what I tried to achieve was very complex and I apparently did very well to get as far as I did without installing onto fresh boxes. I'd only been a Systems Admin for about a year at that point, so I was quite pleased with what I had achieved.

I learned a few lessons from that which have proven invaluable -the main one being "plan for everything". There was no way I was to know I would end up screwing up a server without the smartstart disk at that point, so that was another good lesson.

It all makes you a stronger engineer

pitviper

Mrock4 wrote: »

And my golden rule is to always be honest when you make mistakes- they actually build your credibility with your coworkers, so that when something breaks in the future, and they ask if you did it..and you say "no" - they'll believe you. So it sounds like you did good.

I can’t stress this enough! Sh!t happens, so be honest about it. There’s nothing worse than a co-worker denying that he was involved – or even worse, wasting time trying to “prove” that it couldn’t have been something they did, rather than helping rectify the issue.

I recently pasted some config changes into a window that I was using for reference... killed MPLS and phones for one of our larger branches (120 phones), and took out the second largest at the same time by pasting in a duplicate IP. Luckily it was at night, and the backup cable VPN connections weren’t effected

Hypntick

Kaseya is pretty interesting like that. One of the junior guys here was pushing out an update to the AV on a clients workstations. He set it for 1 pm instead of 1 am, they were a little ticked about that one. But then again it happens, he just needs to pay better attention next time.

networker050184

Worse one I ever did left over 20,000 customer without service. Oops! Have messed some smaller ones taking down ~1,000 customers too. It happens just don't let the same thing happen twice!

shodown

networker050184 wrote: »

Worse one I ever did left over 20,000 customer without service. Oops! Have messed some smaller ones taking down ~1,000 customers too. It happens just don't let the same thing happen twice!

Dead.

I'm loosing count as well as I get older.

bobloblaw

Gotta own it. The person who messes stuff up and doesn't own it - STAY AWAY. The same person is a step away from putting something on you or others.

That said, very early into my career I was phone support for "Road Runner" on Time Warner. Long story short - the button to bounce a node was next to bouncing a user's cable modem. Yeah... guess which one I accidentally clicked on one time? Needless to say they changed their interface within the year. I clearly wasn't the only one that did that.

DB Cooper

I call that a Tuesday for me. Live and learn, mistakes make us better when dealt with the right way.

emerald_octane

Just own it. We all make mistakes. If one of my people blew up a critical box but owned it and showed remorse, I would probably forget about it after a while. If they didn't give 2 craps about the chaos that was caused and tried to roll the blame, that would be a Resume Generating Event (not my term lol) for sure.

JustFred

It happened to me at a former job a few years ago when i was a windows admin, luckily the boss was cool and everyone in management said it was an honest mistake. But yeah i didn't sleep for days after that incident. I still have a drink with the boss every now and then

.

Honestly every IT engineer goes through something like this at some point in his career, its what makes us good engineers

f0rgiv3n

Did the GPO thing... as a company (i know, bad idea) we linked all GPOs to the domain... I was testing and creating a completely restricted policy for kiosk machines and forgot to remove "authenticated users" from the GPO. After creating I left for the day. So I ended up hiding the C drive, no menus on anything, no saving to the desktop, no icons on the desktop, changed the wallpaper and no right click menu on all users that logged on from the time i created it to the next morning when I got in and saw the mayhem.

To this day I'm sure someone still has that wallpaper on their monitor... It was some random landscape pic I found on the web. Good thing it wasn't some sort unprofessional wallpaper.

Oh and your rebooting reminds me of a time when I was with my boss... We were working on a workstation and he had RDP'd to our PDC (THE ONLY domain controller). We finished what we were doing and he goes "Start-> shutdown -> *enter*" he missed the log off button and clicked shut down.

As soon as he did it we both looked at each other O_O and ran to go turn it back on.

Mrock4

A friend/coworker of mine once hit the emergency power off (EPO) button in our data center which powered down 100% of everything..........twice.

Nothing happened to him, they were more angry at the brainiac who placed the EPO button directly behind the door.

CodeBlox

LOL! To assure me that everything would be OK, the director of IT shared with me his big oops and that was exactly what he did too!

About7Narwhal

Mrock4 wrote: »

Nothing happened to him, they were more angry at the brainiac who placed the EPO button directly behind the door.

Ours is on the wall with the rest (FM200 and Fire), surrounded by Yellow on Black tape and covered by a plastic spring door with a second lifter inside. Lifting the spring door activates an EPO strobe. Anyone who set that off would certainly do it on purpose. Here is the kicker though; the door release to leave the Data Center is in between the FM200 override and the EPO. I have to assume at one point they were not covered and accidentally hit instead of the door release.

QHalo

Had someone do the exact same thing. Didn't get let go either.

Mrock4

This individuals "punishment" was to build a box to protect the EPO button..I thought that was very fitting.

I also one time was configuring a switch (core VSS switches) and had everything drop..thought I did it..ran into the data center, and this goofy guy was standing there and said "I think I hit that button..." (it was the EPO..different guy) - classic..

About7Narwhal

I have always wondered what it would be like if someone hit it. I imagine it would be kinda surreal, such a loud room being so quiet all of a sudden.