Options
Mistakes you've made on the job
RHEL
Member Posts: 195 ■■■□□□□□□□
Just curious to see what type of goofs everyone has made in the workplace. Were you new? What was the impact? How was it handled?
I'll start... First few months out of undergrad and working for a large aerospace/defense company, I'm for some reason immediately given the keys to the kingdom (r00t) without hesitation. I'm very tired this day, having been out the night before celebrating St Patty's day, and am writing a bash script on one of the large dev ERP servers. I intend to create a sleep/pause function in the script, but for some reason use the command 'halt' instead. I test my work and guess what? Boom, server goes down.
It took me a second to realize what I had done... It was immediately confirmed as my ssh session was terminated and the server was no longer responding to pings. While my team, team lead and manager worked in a different state, I rushed over to an on-site UNIX manager of another team and let him know what I believe happened.
I ended up notifying my team lead of what happened -- pretty sure I received the idiot of the month award for that one. It was well taken, though he let me know they had noticed and were investigating the cause of the shutdown as I messaged them. My mistake was never forgotten though; when I eventually got to take a trip up to their office and meet the team, a nameplate has been created for the cube where I would be working with the label "Halt" printed on it.
That's about it for big mistakes in my last 4-5 years. I've seen others make big mistakes (taking down networks), and I've seen people fired over mistakes. At my last company, I was quite literally in charge of an AIX-based casino gaming system that required uptime to take in guest money (100+ million a year). At large global companies, I've certainly been in charge of systems as important, though I'm pretty sure an oops at the casino would have been a no warning, you're fired. That's just how it went there.
I'll start... First few months out of undergrad and working for a large aerospace/defense company, I'm for some reason immediately given the keys to the kingdom (r00t) without hesitation. I'm very tired this day, having been out the night before celebrating St Patty's day, and am writing a bash script on one of the large dev ERP servers. I intend to create a sleep/pause function in the script, but for some reason use the command 'halt' instead. I test my work and guess what? Boom, server goes down.
It took me a second to realize what I had done... It was immediately confirmed as my ssh session was terminated and the server was no longer responding to pings. While my team, team lead and manager worked in a different state, I rushed over to an on-site UNIX manager of another team and let him know what I believe happened.
I ended up notifying my team lead of what happened -- pretty sure I received the idiot of the month award for that one. It was well taken, though he let me know they had noticed and were investigating the cause of the shutdown as I messaged them. My mistake was never forgotten though; when I eventually got to take a trip up to their office and meet the team, a nameplate has been created for the cube where I would be working with the label "Halt" printed on it.
That's about it for big mistakes in my last 4-5 years. I've seen others make big mistakes (taking down networks), and I've seen people fired over mistakes. At my last company, I was quite literally in charge of an AIX-based casino gaming system that required uptime to take in guest money (100+ million a year). At large global companies, I've certainly been in charge of systems as important, though I'm pretty sure an oops at the casino would have been a no warning, you're fired. That's just how it went there.
Comments
-
Options
Shdwmage
Member Posts: 374
I changed the time of the primary domain controller and brought my entire network to its knees. I intended to schedule the time change, but I put in AM instead of PM and at 10:00 AM no one could connect to the server anymore.--
“Hey! Listen!” ~ Navi
2013: [x] MCTS 70-680
2014: [x] 22-801 [x] 22-802 [x] CIW Web Foundation Associate
2015 Goals: [] 70-410 -
OptionsNinjaBoy
Member Posts: 968
My biggest "mistake" was deleting the finance database during a financial audit about 8 years ago - that was fun...
This was pre-SQL. We migrated the finance database from an old failing server onto a new shiny server, everything was moved across... Part from one little pointer (which I would like to say, weren't told about). We left the new server running for a few weeks to make sure that there were no problems then formatted the old server.
Because of this little pointer, all transactions were still being written to the old server. We could have restored from backups, but the last time the old server was backed up was a few weeks before as the new server was being backed up.
All I can say is that with a lot of luck and a whole load of data recovery skills and programs, we were able to recover the whole database with all the latest transactions. It took 1/2 day, but it saved a lot of work by the finance department and us, the IT department.
We learnt our lessons and in turn implemented safeguards to avoid things like this happening in the future