1st day on the job - don't be this guy

https://www.hackread.com/man-accidentally-destroyed-production-database-on-first-day-of-his-job/

HA! That's pretty crazy having access to production db - the ID10T error is definitely within their security team when giving access to their network. He should probably only have access to the test db until they can prove their skill set with working in db's.

Cheers & HI5!

Find more posts tagged with

Comments

NetworkNewb

hahahaha nice. Yea, I don't even blame the guy who made the mistake. First real job, following instructions, accidentally put in the what the instructions had listed instead of switching his own info into a line.

The idiot that made those instructions and kept that info on the instructions sheet should be the one who got fired.

Funny story though!

keenon

wow, that is crazy. I do agree they both are at fault.

Company Fault
1. They left the guy by himself except with the guidance of a document.
2. They left prod stuff in the test environment setup doc

3. They also gave him full unrestricted access on day 1 ( when i used to be the manager, new techs got basic but nothing until I was comfortable with them understanding the environment)

His fault
1. Not following directions
2. Not asking for help

greg9891

Wow that's a huge Gamble.....lol thats what you call a real cautionary tale......poor guy.

NetworkingStudent

Why didn't the company have a backup of the production DB?

volfkhat

Sounds like they DESERVE to be outta business...
(almost

stryder144

My favorite response: "What was their tech stack, Jenga?"

blatini

Great story. Everyone makes mistakes - definitely on the company to prevent that.

TechGromit

shochan wrote: »

HA! That's pretty crazy having access to production db - the ID10T error is definitely within their security team when giving access to their network. He should probably only have access to the test db until they can prove their skill set with working in db's.

This is a documentation issue, no one should be issued documents first day on the job, experienced or not, detailing production configuration information. The company would have a difficult job winning a case in court for damages in my opinion, this was an accident waiting to happen, if not this new hire, then the next one. Also read only access might have been a good idea for someone starting on there first day.

greg9891

The company is 100% liable. Having backups that didn't work is a failure of all levels on their disaster recovery plan. That person should definitely not be the only person that should have been fired that day. All he did was expose the recovery and disaster recovery plans did not work. He exposed that their policies are not up to standard. the role of least responsibility wasn't in play at all. This company Looks like it got what it deserved.

TechGromit

greg9891 wrote: »

The company is 100% liable. Having backups that didn't work is a failure of all levels on their disaster recovery plan.

Unfortunately deficiencies often are overlooked in most organizations until something blows up. Backups should be tested on a quarterly bases to verify they are functioning correctly. When I did backup / server administration, I had to restore one file from a backup each quarter to verify the backups were functioning as expected.

mzx380

Poor kid just got out of college and messed up on his first day. The sad thing is he could have been a star by exposing shoddy practices of an unprepared organization. In the long run, this will be to his benefit because he likely join a better organization that will polish him technically and professionally. The higher you start out the gate, the better your career will be.

yoba222

Dang the original Reddit post is locked now. Wanted to see a happy ending play out over the next few weeks where the guy is successful with a wrongful termination lawsuit and where the CTO gets fired.

TheFORCE

That is hilarious. CTO is in for a rude awaking, he should start looking for a new job too. How can you not be backing up a prod database? Insane. How can you create documentation using prod as examples lol.

SpetsRepair

Jesus, can't believe places like this still exist in this day and age with tech the way it is this company must be a failure from the top down. I had to beg to get access in my first 2-3 months and let alone someone has to walk you through accessing certain networks, systems, apps etc....

UnixGuy

Company who doesn't have backup...had a document that includes prod details instead of Dev...gives prod DB access to a new hire on their first day...their CTO want to 'involve legal'...

If I was the CEO, I'd let the CTO go, and bring that new hire guy and apologise to him for the all stress caused to him.

DatabaseHead

It will be interesting if they actually do come after the guy. I think that's a big fat no, but you never know.

I work with data all the time and believe me too much access is a bad thing......

Scary

Danielh22185

Poor guy. He made an honest mistake and hopefully nothing comes of it.

I put this ALL on the company. Why on Earth would someone be tasked with doing something that hold such a giant failure potential on day 1. Sounds like he was simply trying to follow instruction. Also it sounds like the company has no structure built into training up their employees. If the company doesn't have time / resources to train (disaster waiting to happen), then they should at a minimum have a strong structure built around their operational documentation.

wd40

May be they didn't give him admin access on the first day, may be the admin user and password were in the documentation that they gave him ..

Mike7

In a similar vein

British Airways says IT outage may have been caused by human error

1st day on the job for the engineer?

anhtran35

CTO should be fired. NO BACKUP???

skswitch

Looks like that DID have backups but it wasn't restoring. So add not testing their DR plan to the pile.

"So I left. I kept an eye on slack, and from what I can tell the backups were not restoring,"

TechGromit

skswitch wrote: »

Looks like that DID have backups but it wasn't restoring.

An untested backup is not a real backup. You should restore at least one file every so often to verify things are working the way you assume they are.

TechGromit

Mike7 wrote: »

British Airways says IT outage may have been caused by human error

1st day on the job for the engineer?

Need more details, not exactly sure how a single UPS is supplying power to the "data center". If it's really a UPS power supply issue, it more likely one rack of servers with only one functioning power supply each that was disconnected. To affect a "data center", you would have to trip the main breaker on a panel box and power it back on, maybe a power surge from a quick off / on on a main breaker panel? Maybe the term Data center is used loosely, one rack perhaps? Why doesn't the rack have redundant power sources and power supplies. The racks I manage, all the equipment except the KVM/monitor have one power supply connected to UPS and the other to house power.

ITSec14

UnixGuy wrote: »

Company who doesn't have backup...had a document that includes prod details instead of Dev...gives prod DB access to a new hire on their first day...their CTO want to 'involve legal'...

If I was the CEO, I'd let the CTO go, and bring that new hire guy and apologise to him for the all stress caused to him.

If they can't get the backups to work...they may just be out of business.

wd40

TechGromit wrote: »

Need more details, not exactly sure how a single UPS is supplying power to the "data center". If it's really a UPS power supply issue, it more likely one rack of servers with only one functioning power supply each that was disconnected. To affect a "data center", you would have to trip the main breaker on a panel box and power it back on, maybe a power surge from a quick off / on on a main breaker panel? Maybe the term Data center is used loosely, one rack perhaps? Why doesn't the rack have redundant power sources and power supplies. The racks I manage, all the equipment except the KVM/monitor have one power supply connected to UPS and the other to house power.

Unfortunately not all equipment have redundant power supplies, and it is possible that at that moment the main power was down and the UPS(s) did not take over (connecting equipment to "mains" directly is not a good idea and Doing a full data center UPS test is not an easy task so people skip it until a disaster happens.

And it is possible because the engineer just switched the power back on in an uncontrolled way, the equipment did not start in the correct order which lead to a chain-reaction that brought the backup site down (and of course the main failed to start correctly) so now both sites are down.

thomas_

TechGromit wrote: »

Need more details, not exactly sure how a single UPS is supplying power to the "data center". If it's really a UPS power supply issue, it more likely one rack of servers with only one functioning power supply each that was disconnected. To affect a "data center", you would have to trip the main breaker on a panel box and power it back on, maybe a power surge from a quick off / on on a main breaker panel? Maybe the term Data center is used loosely, one rack perhaps? Why doesn't the rack have redundant power sources and power supplies. The racks I manage, all the equipment except the KVM/monitor have one power supply connected to UPS and the other to house power.

You might have too narrow a view of UPS systems.