Stupid Things You've Done?

dstock7337

RobertKaucher wrote: »

That sucks! "Windows Key + L".

I can only imagine the sinking feeling... Was the VP aware of what was going on?

No. Don't think he cared about what happened as long as everything got back up quickly. I had already given a quick brief to my manager right off the bat and a detailed explanation after fixing the problem. Honesty (and having good backups) go a long way.

swabbies

We were in the middle of a server room expansion project. We had about 8 hours to physically move the core switch (6500) and about 100 servers. They only had to be moved a distance of about 40 feet, but it meant recabling the entire datacenter. To make the core switch lighter we removed the line cards while we carried it. I was trying to move a little too fast and didn't quite line up one of the 48 port cards correctly as I slid it into the chassis. I pushed with a bit of force at the end to make it fit, and then I heard the sound of plastic breaking. The backplane of the chassis and the linecard were both toast.
Thank god for Cisco 4 hour support. I think we made the replacement and recable with about 20 minutes to spare. We have learned since then to make sure we have a redudant core switch.

arwes

The tech I deal with from the company who made our insurance software had me delete & recreate a (particularly obnoxious) remote user's Exchange profile on the terminal server because she was having trouble attaching certain emails to their software. I had no idea this person had been relying on "cached" email addresses to correspond with people outside the company. Needless to say, those were wiped out and she went ballistic on me.

I immediately went to my boss about it (and she to hers), and thankfully he remembered the email our CEO sent out when we went live on the new software. He basically told everyone that if they've got an email address they want to keep, they'd better add it to their contacts because it won't be saved any other way.

Another story relating to the same software. We were having problems with their document imaging portion of the software, and were backed up on about 3 weeks worth of scanning. People were needing access to the documents, and they were in a box on someone's desk because the app kept crashing. I finally got a tech to remote in to help me with the workstation, and a new manager walked in complaining about her phone not working during a conference call. I told her I'd be in there in just a few minutes to look at it, as soon as I was finished with that workstation. Her response? "I think this is a little more important than whatever it is you're working on." My response? "No it isn't, people even outside the agency are screaming at us to get this fixed NOW and I don't know when I'll ever be able to get another tech to look at this."

She stomped off down the hall and I heard her slam her phone on the desk a few times (oh I wish she would've broken it) and then it miraculously started working. Ever since then, she's tried to find something to complain about me at the weekly manager's meeting but my boss just brushes her off. It's funny too, the other manager is her polar opposite. We've got one that's high strung and takes everything personally and the other is completely laid back without a care in the world. Guess which one the employees like best?

eltoro

Watch **** at work.

NetAdmin2436

undomiel wrote: »

Back in my early administrative days I implemented ipsec on the sole DC. Which of course resulted in enhanced security of it being unable to talk with any clients or other servers. Fortunately I caught that really fast and was able to sweep things up before the locals caught on.

I did the same thing in my first year as a netadmin. I thought I'd be cool and try and increase security on our DC, even when I knew nothing about ipsec at the time. Good times, good times.

I also uninstalled iis from our exchange server. Exchange was down for the entire day and it was a pain in the arse to fix it. I was the only IT person at the company, so naturally I told the owner of the company that it was simply a bad windows update that caused exchange to crash. He bought it

Ryan82

I was sitting at one of our servers console and was RDP'd into our exchange server. During the midst of troubleshooting 3 things at once I accidentally shut down the nic on our exchange server, which of course took down mail for everyone. What made it worse, is that we don't have the password to log on locally to this machine so we had to wait for 4 or 5 hours for our higher HQ admins to come in stateside and give me the password to re-enable it.

My old supervisor and I were replacing some switches in one of our racks. Though we had talked about whether or not we should shut down power to the other devices in the rack, we decided that a simple 3750 switch replacement wouldn't require shutting down the servers in the rack. Well, in the midst of replacing one of the switches his belly hit the APC power switch and killed power to our DC's and exchange server. No big deal right? Well, after we powered back on the exchange server, it turns out the pub and priv databases were corrupt. To further add insult to injury, our backups had been having 'issues' and so our most recent solid backup was from over two weeks ago. I spent literally 26 hours straight troubleshooting this thing to get it back online.

Forsaken_GA

Ryan82 wrote: »

Well, in the midst of replacing one of the switches his belly hit the APC power switch and killed power to our DC's and exchange server.

That sounds familiar. Our old data center used to have our servers racked in two post relay racks. One set of racks was particularly close to another set, creating a very tight fit. I had to shimmy in to hook up a head to a server, and I'm not a small guy. Apparently my fat ass hit the power switch on the PDU behind me and took down the entire rack of servers. I was oblivious until the NOC called over and asked wtf I'd done. It was my first week on the job.

Fortunately, nothing major occurred, the servers all came back up without issue, and I learned a valuable lesson about paying attention to where my body parts were while I was in the data center.

I do find it somewhat amusing that there's a high number of stories that all relate to windows domain controllers or exchange servers. Let's here from some fellow Unix nerds about their major screw ups.

I thankfully have not done anything major to piss a customer off, but a coworker made a mistake that was another good object lesson. Monitoring alerted to a server having a full disk. He gets on the server and deletes the mysql bin logs.

Well, it turned out that there were a pair of database servers that were setup in master-master replication, and one of them hadn't read the logs that were just deleted. This, of course, hosed replication entirely. This was from a very high traffic adult site, and that database happened to contain their billing info. So the databases were totally out of sync, and both had information the other didn't. Needless to say, that one was an emergency escalation, and from what I understand the tech in question got a serious tongue lashing from the president of the company. The rest of us got a nastygram about how we weren't going to do that again. Ever.

msteinhilber

Forsaken_GA wrote: »

I do find it somewhat amusing that there's a high number of stories that all relate to windows domain controllers or exchange servers. Let's here from some fellow Unix nerds about their major screw ups.

When I was doing my own web hosting biz I on a couple of occasions botched things up when I was deploying a new server. The one that seemed to get me was when I would configure APF and I would rush through things and configure the ports and disable DEV mode in one shot and reload the firewall. A couple times I must have been tired (web hosting was a side biz for me so I often performed these tasks in the evening to late night hours) and spaced out and opened port 22 for SSH instead of the alternate port I always use. Then it was a decision to either pay for remote hands at the data center I co-located at or drive there myself. Fortunately it wasn't much of a drive for me so I usually just opted to drive down since they were new deployments and hadn't actually had any customers hosted on them at the time.

That's about it for my big goof ups on Linux other than an occasional careless rm -rf but those were usually easily fixed from a backup.

SysAdmin4066

A few years ago I approved an update for SQL on WSUS (one that I hadnt vetted) that resulted in the server crashing, blue screning and never coming back up. It had to be rebuilt, recovered and it took the entire 8 hour day to do. This server held financial info so needless to say I was holding my breath during the rebuild process

ColbyG

msteinhilber wrote: »

That's about it for my big goof ups on Linux other than an occasional careless rm -rf but those were usually easily fixed from a backup.

I rm -rf'ed something a long time ago, not sure what string I used but I executed it from / and it basically made the box unusable. Thankfully it was one of my boxes at home and not something in production.

Mrock4

I posted it somewhere on here..but here's the quick rundown:

I was new at the time to foundry devices, and entered port sec from the global configuration mode (instead of the interface mode)...didn't realize I was configuring for all of the ports on the buildings distro switch...then I enabled port security on all ports mistakenly (including the trunk). I casually told my co-workers I'd be back in a few minutes. To save time I just rebooted the SW restoring the config..naturally, this made the entire building of users to be completely in the dark for a little while. Oops!

GT-Rob

I got one that just happened 5 mins ago (during a change freeze to boot!). Good thing I was 10feet from the cabinet.


A switch stack with about 80 or so users on it with a single uplink to our core had bpduguard and bpdufilter on it. I thought "that's silly, ill take those off".



I took filter off first....

skrpune

my most recent DOH! moment was during a desktop reimaging project, when I deleted user accounts instead of the computer accounts. That was an embarrassing phone call to have to make "Um, hi...I'm a jackass and I deleted a user instead of the computer...yes, I know it was in a different OU...sigh..."

contentpros

odd that I see this thread now....

I just spent the morning cleaning up a jr. tech's mistake... he saw an alert for one of our circuits flapping and took the initiative to rearrange some cards in our Optera 3400. For a large call center this is bad luckily since we are on holiday hours the impact was minimal (thank god for backup voip routing!) if not for the tech calling me in tears and taking ownership for the error this probably would have been a resume generating event.

dynamik

I just introduced my keyboard to 16oz of Redbull-Vodka. The worst part is, I was only a couple of sips in, so it just my natural clumsiness. That's why I only drink from the safety of my couch...

Met44

Forsaken_GA wrote: »

I do find it somewhat amusing that there's a high number of stories that all relate to windows domain controllers or exchange servers. Let's here from some fellow Unix nerds about their major screw ups.

For anyone who has not read "The classic UNIX horror story": *THE* classic Unix horror story | WSU Linux Users Group

Very old, but a very good story.

dynamik

Forsaken_GA wrote: »

I do find it somewhat amusing that there's a high number of stories that all relate to windows domain controllers or exchange servers. Let's here from some fellow Unix nerds about their major screw ups.

Whoops, missed this. So back when I used to hang out on IRC, someone asked how to defrag a Linux system. I said I didn't know off the top of my head, but I think rm -rf / will do something similar. I thought the joke was obvious and didn't give it another thought. A few seconds later, the guy was like, "What did you have me do!"

Forsaken_GA

dynamik wrote: »

Whoops, missed this. So back when I used to hang out on IRC, someone asked how to defrag a Linux system. I said I didn't know off the top of my head, but I think rm -rf / will do something similar. I thought the joke was obvious and didn't give it another thought. A few seconds later, the guy was like, "What did you have me do!"

I have a very amusing anecdote about why you don't give your customers root access if it's a managed machine.

One guy managed to talk our management into allowing him to have root to set a few things up 'his way'.

A few days later, the machine starts popping errors left and right. Try to ssh in, and it bitches about ssh exchange identification (which is almost always a good clue that /var is fubar).

So I go console into the machine. 'Someone' had done a recursive chown to the customers username. From /.

First thing I did was edit /etc/sudoers and pull his access. That recovery was not fun at the time, but it's amusing now.