Have you ever lost user data?

G.O.A.T

One of the biggest concerns I have is losing user data and I often backup regardless of being told I dont have to. Im wondering have you ever lost critical data whilst at work? and if so what was the outcome?

tmtex

Nope ! Always copy off files before a reimage, better yet if you can, change HDD's.

TechGromit

Yes and No. I was in charge of a mailbox client upgrade from Novell Groupwise to Microsoft outlook, I was working the swing shift overseeing several outside contractors assisting me. One of the computers we worked on, all the user files including all the customized forms files were lost, the user stored all there forms locally, even though network drives were available. The department manager made a big deal about the lost files, since they needed them to respond to customer data requests. We tried to recover the files, but I don't know what was done to the computer but they were unrecoverable. At the end of the day my manager told them tough, they should have been storing critical files to the network file server where the drives get backed up daily.

clarson

like 20 years ago, worked at a data center for a fortune 500 company. They had a computer go down at a manufacturing plant. Went out to the plant and fixed it. Then, went to restore the data. They made a back up every night. On the one tape that came with the system when it was new. They didn't get their data back.

NetworkNewb

Ohhhh yea, didnt backup a users data correctly a few years back while I worked at help desk, reimaged their machine, bye bye data.

I let them know what happened, they were pretty pissed. I let my mananger know, what I did and where I screwed up. Not much else to it... Manager wasn't too happy and user had to do a bunch of work to remake the stuff that was lost. It is what it is. Just can't let that happen multiple times.

Mike7

In a previous engagement, management refused numerous requests to buy 25k USD network backup system so infra team backup to another disk partition on the server. The server was part of a system for external customers that generated north of US 7 million revenue annually.

Both disk data and backups were corrupted one time and infra was unable to restore any data.

Discord

G.O.A.T wrote: »

One of the biggest concerns I have is losing user data and I often backup regardless of being told I dont have to. Im wondering have you ever lost critical data whilst at work? and if so what was the outcome?

Yup, though I can honestly say that none were a direct result of my actions. All have been due to the backup hardware failing to some degree and replacement units or parts never being purchased - both due to lack of funds or interest (hey, everything still works and there's RAID arrays right?).

#1 Have a backup and restoration plan and test it at least quarterly. No exceptions.

#2 If there's an issue, management/supervisor not listening, have a message trail noting that you stressed the failure and need for replacement.

#3 Repeat yourself and your concerns and add to the message trail as much as needed. Archive any responses you get. If you are ignored, mention in follow up message traffic (ie: I didn't get a response, so I'm not sure if you got my previous message).

#4 If/when something happens, whip that **** out when UPPER management asks "well WTF happened, we have a backup system!". Don't be afraid to interface with the suits and speak your mind (professionally of course!). State facts, keep opinion out.

#5 Never say "I told you so". The message traffic will speak volumes.

#6 Go to bed and sleep well that night. You tried, they did not listen.

blargoe

Yes, I have.

jeremywatts2005

There are times in security where data will never be recoverable. I know of malware that can infect the entire OS and then spread to the users files. Even scanning the users files does not always throw a flag as to the infection being there until it starts to spread on restore and do C2 call outs. Usually the user launches a document or a file that was actually created from the previous infection that looks like a legit document. So data loss is a fact of life sometimes.

LeBroke

It took me a few seconds to realize you don't mean the user_data script and I was wondering why it matters if you lose it since it's auto-generated upon instance launch...

brewboy

Luckily no. One reason I like network is no worry of data loss

skswitch

Ha! Oh boy do I have a story about this one.

First IT job doing help desk, first major project. It was a multi domain environment with accounting being one side and wholesale being another. Why it is like that is another story but its what I walked into. Project is Upgrading / Virtualization of Exchange that hasn't seen any love since its been put in place.

Well in this environment our wholesale branches are bad about not logging into there account using a generic branch account, sharing password and not even being told a person is hired for account creation. I know...another story and project for later.

We start off by a purge of seeing what mailboxes we should leave behind. Ones that were not active by 6mths+ was set criteria by CTO. Terminated accounts that also have no policy on what to do with no old data. So..if you haven't guessed already with people not logging into their accounts combined with leaving behind inactive accounts... some were active! This is also after numerous warnings sent to all users to use their second domain account before project even started.

I get a call from first user who is saying they don't have any email. Go figure, he has no email account. Not telling him this I try and find a way to get something at least back. Now forgive me if I get the terms wrong on next part, I haven't done any admin work with Exchange since that job which was 5+ yrs. What I was able to do was use his cache profile that he had stored locally and add it back as an archive. Then create a new mailbox with his user account and attach that archive so he could access it. User complains of course in the end cause its a extra step and different from what hes use to. Be happy you got anything!

All in all we were able to migrate 2000+ mailboxes leaving ~1000 or so behind. Less than twenty people had any issues which we were able to replicate what I did with cache file. Leaving us with 98-99% success rate.

User did email IT though a nasty gram that my CTO answered. He responded to me how professional I handled it and a *nod* of approval. Lesson learned. No matter the heat, handed it with grace and don't let emotions get better of you.

MAC_Addy

Yup. Not only have I lost one person's data, I have lost the whole "corporate" peoples data. I was brand new to a position and didn't know a whole lot about the environment.

It sounds bad, and it was a nightmare for about 4 weeks. We had a server with 3 drives in RAID 5. One morning, I came into work and found the screen erroring out on many, many things. Rebooted, same thing. Got on the phone with Dell support. Found out that one drive, potentially two drives went bad. We had 4 hour parts support, so it didn't take long to get the drives out to us.

We get the new drive installed and running through the rebuild on the actual bad drive. After a couple of hours it was rebuild and needed to be restarted. Went to boot, same errors.

Turns out that one of the drives had been going bad for a long, long time. Long enough for RAID 5 to copy over errors and bad sectors to the other drives, rendering them useless. We didn't think much of it since we're backing up via tapes every night. The schedule was: Full backup on Monday night, starting at 8:00PM. Incremental every other night after that, apart from Saturday's.

Got the server back online, configured AD, DHCP, and DNS. All was well from this point. Theeeeen we started to restore "data". Reloaded Backup Exec and started the process. It kept on asking for the sequence of tapes, and then started to do something weird - it kept on asking for one tape in particular, then another after about 2 minutes.

Since nothing else went our way during this process, we found that the "user data" folder was missing, along with legal files, HR data dating back 3 - 4 years. Yeah, it was BAD!

What I had everyone do in this time was search through the old e-mails for attachments. Mostly everything everyone ever sent was in there. We didn't have any retention policy, so it was easy to get some back.

After this, we got rid of those bloody tapes! We moved the backups to go through a MSP with bare metal backups. Lesson learned. Always check your backups.

cyberguypr

If you ask me I would say I've never lost any user data. However, I worked with a lady that would disagree. This was back in 1998-99. The user complained about her machine being slow. My first step was checking disk space and memory utilization. All looked good. Second step, emptying the Recycle Bin. The minute I did that the lady yelled "NOOO!!!!". I didn't immediately understand why she was yelling. When I asked if everything was OK she said "Why did you do that? I keep all my archived stuff in the recycle bin so it doesn't fill my disk." True story. I was lucky we had strong policy indicating everything important had to be saved to H: and no one was responsible for stuff on the local drives. BTW, this is the same lady that said that files would copy faster if the icons with the flying paper were closer together.

NetworkNewb

LOL@cyberguypr! Love that. A couple years back I had user who had a habit of storing their old emails in their "Deleted Items" folder in Outlook...

Seriously people, do you store your hard copy of files in the trash bin under your desk too??

TechxWizard

This happen so often. Especially at local repairs replaces. I use to work during my college years for a place that sells office supplies....I was an easytech (thats what they call it) It is very easy to get the job because the exam you take is similar to the A+ and its unsupervised. ( meaning everyone uses their phone to google answers ). It never happen to me when i lost data, but I've seen guys trip over sata/usb cables and drop hard drives left and right. Of course before you leave your laptop you sign a disclosure stating were are not liable to any data lost. At that point if you really want your files, you pay $1,500 and we ship it to a lab to retrieve your data. Whose to say drives werent dropped on purpose to drives repair sales... Ive learned alot there and seen it all... So glad Im not there anymore