"Oh God Why" Moments~

Sevenplyheart

These are awesome.

Not too soon after I started my current job I was applying a newly assigned static IP to my machine and fat fingered a number. I noticed at soon as I did it and corrected the IP but not quite quick enough. The next thing I heard was "Why is blah-blah-blah filer down?" followed by "Why is it reporting a duplicate IP?"

Needless to say I double check all IPs before hitting enter now, and am still called "el' dupe" by my co-workers almost a year later.

Kelkin

Lets see.. When I first got into networking I didnt put permit ip any any in an ACL and took down an entire campus network..

Good times..

elTorito

- Reverting a production server to a previous VMware snapshot the night before it was supposed to go into production, instead of committing it. I was able to restore the server from backup, but the sinking feeling you get after pressing the wrong button...

- Deploying a replacement to a MSI package via Group Policy, which failed miserably because the previous package couldn't be uninstalled gracefully. The end result was that neither package worked properly afterwards. However, the next day 40+ people needed to use that application, so suffice to say I spent the whole next day manually fixing things at each user's desktop, because I didn't have enough time to concoct a scripted fix. That was fun!

- Switching off a user's desktop because I thought it wasn't in use, when in fact that user was sitting at her colleague's desk a few feet away. I don't know if she lost a lot of work, but that was definitely an 'oh crap' moment.

- Causing a short circuit after plugging in a faulty power unit. At least 10 users' desktops went out, including my manager's. No one held it against me, but even now people are still wary when they see me handling a power cord

demonfurbie

once i defraged a live database server.....


yea that will never happen again

badrottie

Thankfully, I have not had a major incident yet, but I have observed some classic ones.

Going into a new data center, and seeing the ESD button at head-height with no physical protection to prevent it from being accidentally tripped. I stated, "Someone should put this at least in an enclosure" and being told "Not to worry". Several months into production and during a maintenance window scheduled at 15:00, I watched a tired Sysadmin lean his back against the wall and hit the ESD with his head. Instant "lights-out" data center.

Same place, but a few months later into operation. I saw that all the power cables for a new bank of racks were plugged directly into the wall outlets, and with no physical protection to keep them from being unplugged, cover them, etc. Gave another warning about this being a "bad idea" for several reasons. Told to mind my own business by the Network Engineer--again. It also did not help that they had the rack enclosures so close to the wall that there was very little working room behind them. Of course, during another maintenance window at 15:00, I watched a Junior Sysadmin lose balance, trip, fall, pull the plugs and de-energize a row of racks. Tried not laugh.

Last one was a beauty. Again, same data center (Whether there is a pattern here or not, I will leave as an exercise for the reader). Because the data center was not a room built for purpose, they did not have cable management to speak of, which is a polite way of saying that it was a rat's nest of low-voltage and fiber optic cables strewn all over the place--including right in the middle of the corridor. Did the obligatory warning, told to mind my business yet again--wait for it--another maintenance window, another accident, and the ISP uplinks, the ties to other campus buildings, and a whole bunch else is now forcibly disconnected from the switches and racks. Of course, nothing is labeled, and even if it was, a lot of the cables are now physically damaged and reuse is impossible.

The Network Engineer was a very smart man in what he did, but from an operations perspective was an absolute disaster. I was surprised that he didn't get canned after the second incident, because his decisions put employees safety at risk. I guess they were waiting for a lawsuit to occur, to be honest.

hackman2007

demonfurbie wrote: »

once i defraged a live database server.....


yea that will never happen again

Haha, that is awesome!

About how many calls did you receive saying the network was slow?

demonfurbie

hackman2007 wrote: »

Haha, that is awesome!

About how many calls did you receive saying the network was slow?

so many i killed the process, even that didnt work... i did what every new IT guy in that situation would do .... blame microsoft server for needing a reboot

lsud00d

As a student worker using Norton Ghost for the first time (TCP/IP P2P), I decided to wing it and !=RTFM.
Well, I ended up doing remote-->local instead of local-->remote, spreading the chocolatey virus goodness to yet another computer...


'shutdown -r now' in linux terminal, only to realize the tab was for a production server and not my local machine...


Deleted a sub-OU in AD with various objects nested underneath because I did not think I was given those permissions...


etc.

Tackle

That shutdown -r command can be a nasty one, especially (In windows) if you do a "shutdown -r -t 1 -f -m [URL="file://\\PCNAME"]\\PCNAME[/URL]". Should have set the time limit a little longer, didn't realize that user came into work early.

I think most IT pros know the feeling you get after you click the wrong button or accidentally delete something. That "OH SH*T!!!" moment.

Forsaken_GA

pham0329 wrote: »

Reading some of these responses makes me feel better about the dumb mistakes that I have made in the past.

Can you do PBR with prefix list? I remember trying that in a lab and it didn't work. I think Forsaken or someone else mentioned that prefix list aren't designed for PBR..or something like that.

I could never find any definitive statement from Cisco saying that prefix lists and PBR are a no no, but yeah, when someone else presented that problem, and I labbed it up, the results were.... not as expected hehe

Forsaken_GA

demonfurbie wrote: »

once i defraged a live database server.....


yea that will never happen again

hahaha

yeah, that's one that goes under the heading of 'stuff ill never ever do again'

Forsaken_GA

lsud00d wrote: »

'shutdown -r now' in linux terminal, only to realize the tab was for a production server and not my local machine...

It is far, far worse when that command is rm -rf * on the production server.

I've never done it, but I've had to cleanup after those who have. I think the worse one I ever saw was an end user we let have sudo on a web server for one reason or another.

he issued sudo chown -R <username>:www-data ./*

From the root directory instead of his home directory. That particular boneheaded move took down his multi-million dollar website and necessitate a full server restore. And we stopped letting users have sudo on managed machines shortly thereafter.

thenjduke

I remember back in my helpdesk days using Norton Ghost and to Ghost over the network. Lets just say when I was downloading the image the network came to a halt except for my imaging going on the desktop. I never used Ghost after that

DPG

I made some changes to our BGP announcements last month that caused a bunch of routers worldwide of a specific flavor to crash.
I'm sure this was a "Oh God, why?" moment for the admins of the affected routers.

nerdydad

I was bouncing a bundle due to a t1 link being down, I entered the shutdown command and immediately realized I was in the customer device and not in our edge device, had to call the customer and have them reboot the router.

Forsaken_GA

DPG wrote: »

I made some changes to our BGP announcements last month that caused a bunch of routers worldwide of a specific flavor to crash.
I'm sure this was a "Oh God, why?" moment for the admins of the affected routers.

Let me guess, you got a little prepend happy

I'm guessing you crashed a bunch of MikroTik's

undomiel

Well let's try a good number of years back when I was a scripting neophyte. I put together a batch file (no powershell yet!) that would create some temp files then issue a del /Q /S /F *.* Tested the script out several times and it ran flawlessly. Set it as a scheduled task and waited for the first run and watched as it kicks off with the default directory of C:\Windows on the company's ERP server. Quickly canceled it and the server had no obvious problems with it unless you checked the event log or tried to add any roles. That was a pretty frightening moment though.

punkisdead83

demonfurbie wrote: »

once i defraged a live database server.....


yea that will never happen again

This one's my favorite. Our office lives breathes and sleeps our SQL boxes.

I was deploying some new Windows 7 boxes. I removed the old XP ones from the domain and renamed the new ones the same name and added them. When I went to reallocate the old machines I had forgotten to reactivate the local admin account. I spent the second half of my day hunting down a utility and trying to get back into those machines.

jamesp1983

These are pretty good:

The top 10 IT disasters of all time | IT At Work | ZDNet UK

Slowhand

Forsaken_GA wrote: »

It is far, far worse when that command is rm -rf * on the production server.

I saw this happen once, at a school. A student enrolled in a UNIX class took down a one-of-a-kind server that had been built many, many years ago from scratch by a development team from UC Berkeley. There was no way to recover it, there was no install media of any kind, so they had to scrap it and set up some FreeBSD boxes for the next semester.

The incident happened when the regular user accounts had been disabled and the admin for the lab had stepped away from his chair for a few minutes while looking for some documentation. The student in question came in, saw a console window open, and thought he'd use his "expert" skills to clean up the user directory; the admin was, of course, logged in as root. I'm not sure what happened to the student, I'm fairly sure some disciplinary action was taken against him for wrecking the server and for using someone else's login.

jibbajabba

Hitting an emergency powercut button in a DC, shutting the whole floor down

(needless to say they had a rubbish design)

xenodamus

I was fumbling around in SCCM a couple of weeks ago, trying to figure out how to re-advertise a MS Office installation to a particular PC.

I accidentally advertised the software to the "All Systems" container, which includes about 2500 desktops and 500 servers. O.o

Luckily I noticed it, and immediately figured out how to delete an advertisement as well. I think my permissions may have been restricted on that box if the install had gone through, lol.

Mishra

I have a great one LOL

I've formatted the main database server's storage for a local mortgage company. LOL. They lost ALL information they put in for that day.

P.S. Don't ever format a Windows box while it is connected to a DAS/NAS/SAN.

I promise and swear I doubled checked that it said "73 gig" when I formatted that drive, I made sure of it. I think Windows just had a good ole time of every partition on the machine.

DPG

Forsaken_GA wrote: »

Let me guess, you got a little prepend happy

I'm guessing you crashed a bunch of MikroTik's

Something like that except it was a much more well known brand.

ImTheKing

Prior to taking a formal job, I co-owned a medium sized sever hosting/colocation company. We were replacing several of our older switches (about 5, all in different racks). I completely screwed up the scheduled maintenance notice and sent it to the wrong clients. Needless to say, the clients that were actually falling under the scheduled maintenance were not happy.

Forsaken_GA

DPG wrote: »

Something like that except it was a much more well known brand.

Oh, if this was recently, then I'm guessing you found the Juniper BGP bug hehe

Moki99

Anyone get fired or almost fired doing some of this stuff? I'd assume some of these are no joke when it happened.

dead_p00l

Forsaken_GA wrote: »

Oh, if this was recently, then I'm guessing you found the Juniper BGP bug hehe

Which Juniper BGP bug? im running MX960's in our core and <knock on wood> havent run into that yet.

cisco_trooper

The one where all the BGP sessions drop. Discovered around the first or second week of November. Level 3 in Chicago got bit by it if I remember right.

Forsaken_GA

cisco_trooper wrote: »

The one where all the BGP sessions drop. Discovered around the first or second week of November. Level 3 in Chicago got bit by it if I remember right.

That would be the one. dead_p00l, if you're running MX's, make sure JunOS is updated hehe