Whats the biggest screw up you have been a part of at work?

I am going to leave details out because this might make national news, it just occurred and its too soon to tell if it was an attack or a "oops".

The issue:
Every device in an organization was re-imaged instantly (device count in the 1000's). Every device.

They are paging every tech they employ in a three states to assist in resolving this issue.

I think this will be one of the biggest melt downs I might be involved in (the resolve process not the "oops process).

Anyone else have a similar experience? How was the problem solved? How long did it take?

Find more posts tagged with

Free for TechExams community: Cybersecurity salary guide

Compare cert salaries and plan your next career move

Button

Comments

networker050184

If you fire every engineer that makes a mistake you are going to run out of people to hire. People mess up, it's human nature. Even the best of the best screw up. Learn from it and move on.

DevilWAH

networker050184 wrote: »

If you fire every engineer that makes a mistake you are going to run out of people to hire. People mess up, it's human nature. Even the best of the best screw up. Learn from it and move on.

I worked on systems where people could quite literally can end up dead if some one screws up. messing up / screwing up is not human nature You don't fire all the engineers who make mistakes. Only the ones that use the excuse that "mistakes are inevitable". You keep on the ones who are embarrassed by there mistakes and change there approach to work to minimize them, who don't just accept that mistakes happen.

tprice5

colemic wrote: »

Also in AFG, wasn't me but several thousand NIPR machines had an SCCM reimage command issued to them, with no profile backup.Supposed to have been to a test OU but went to all of camp eggers. They pulled techs from all over the country for a few weeks to get everything back to normal.

I've been working at the SWACC for the past year. I am positive this incident predates me but I have a good idea of who it was.

networker050184

DevilWAH wrote: »

I worked on systems where people could quite literally can end up dead if some one screws up. messing up / screwing up is not human nature You don't fire all the engineers who make mistakes. Only the ones that use the excuse that "mistakes are inevitable". You keep on the ones who are embarrassed by there mistakes and change there approach to work to minimize them.

If someone isn't working to minimize mistakes that is a completely different story. Things happen though. Never has or never will be a perfect human.

DevilWAH

networker050184 wrote: »

If someone isn't working to minimize mistakes that is a completely different story. Things happen though. Never has or never will be a perfect human.

What do you mean? there's one right here dude!!!

networker050184

Ok, guess there are two perfect people but we can't do everything!

tprice5

Hey DevilWAH, I see that you are in the UK. I just flew from the states to the middle east with a long layover in London. Got a chance to trod around a bit and see some sites. Really liked the culture and think I would like to maybe live and work there for a year. Do you work with any Americans? Is immigration easy? Is it a hyper-competitive market?

Thanks!

datacomboss

DevilWAH wrote: »

I worked on systems where people could quite literally can end up dead if some one screws up.

I work in a similar situation and I've seen people die as a result of mistakes and yet the mistakes keep happening.

DevilWAH

tprice5 wrote: »

Hey DevilWAH, I see that you are in the UK. I just flew from the states to the middle east with a long layover in London. Got a chance to trod around a bit and see some sites. Really liked the culture and think I would like to maybe live and work there for a year. Do you work with any Americans? Is immigration easy? Is it a hyper-competitive market?

Thanks!

I know a few from the other side of the pond who work over here. But i don't know what immigration is like. I think from the US it is not to bad though as long as you bring skills.

as for competitive I don't know either, I work in a niche area so for me its not about competition, when i worked in prvt sector it could be quite cut throat. But in my current role it was word of mouth that got me in.

Sorry I can't be more help, all I can say is I think this country is great, I did almost move to US about 7 years back, but then meet my wife and had a very good reason to stay here

EspiOne

I have done (freelance) computer repair/OS installation for about 10 years, before I went to school and got my first MS certification. Normally, when I did a OS repair, I would back user data, (copy user documents and settings, I did not know any better) and re-install the OS/drivers and so forth.

Been working Professionally since Dec 2010

My first PC replacement at work: User worked in HR, PC was giving him issues, I got the PC up in my area, copied data to external drive with SATA drive dock, I was told to clean the original drive and re-use, I would used diskpart and cleaned the drives and in this case, I cleaned the external drive also. Lost all the data, as I stood there in complete shock, I told my supervisor, he basically said, "I hope he had a back up, because he PC's hard was bad and finally failed" and we left it at that.

Second Major Screw up. I deployed (shipped) 15 PC to a field office with no static IP addresses, (but I labeled them with IP address), I had to give instructions to the field office supervisor on how to add an IP, not to mention the administrator PW.

Since then, we (I) have developed a checklist for each and every PC/notebook that leaves our division.

Little forgots here and there, but nothing as devastating as these two.

colemic

And to add to my list (and this one was on me), I zeroized a 4-star's TACLANE once. (Encryption device). It's wasn't the old style, it was the new, form-factor one. The old one had a great big sticker warning you to not pop the (I think) AA battery out, but the new ones had no such sticker. Sitting at a network engineer's waiting for him to get off the phone, I was looking at it, saw the compartment, (and no sticker) and mindlessly popped the battery out. Had to send it back to the factory to get reset. Net engineer saw me do it, and I can still see the look of horror and panic on his face when he realized what I did, LOL. Totally my fault, even though it should have had a warning sticker. (I wouldn't have done it if it did, for sure)

--chris--

colemic wrote: »

And to add to my list (and this one was on me), I zeroized a 4-star's TACLANE once. (Encryption device). It's wasn't the old style, it was the new, form-factor one. The old one had a great big sticker warning you to not pop the (I think) AA battery out, but the new ones had no such sticker. Sitting at a network engineer's waiting for him to get off the phone, I was looking at it, saw the compartment, (and no sticker) and mindlessly popped the battery out. Had to send it back to the factory to get reset. Net engineer saw me do it, and I can still see the look of horror and panic on his face when he realized what I did, LOL. Totally my fault, even though it should have had a warning sticker. (I wouldn't have done it if it did, for sure)

HA! Seems like a flaw in the design honestly. If its easily popped out, what keeps a random accident from jarring it loose?