"Oh God Why" Moments~

darkerz

I just sent out an email to every user in my organization, part of our System Admin set of "This is now working again" emails.

I misspelled the word Monday and Resolved. My first mistake at my job was not a technical one, document issue, or downtime crisis.. It was basic English visible by 5,000 people.

Anyone else have similar blunders? My manager laughed it off, saying "I was beginning to get worried, sometimes you do things so smoothly I forget you exist". (Er, thanks?)

effekted

I haven't made that mistake yet but I am the person that proof reads several times when I send out company wide emails. I wouldn't sweat it though (especially since your manager got a laugh out of it and made the comment), I know in my organization I see typo's in emails sent out to the company all the time and etc.

Lizano

oh I've killed live production routers (only once), I friend of mine thought he was in the remote site's router and ran a debug ip eigrp, he was really on the core router which had 1000+ eigrp neighbors, needless to say, the router crashed.

Forsaken_GA

I've brought an entire ISP down before (they neglected to tell me some very important architecture information)

Slowhand

I had quite a few of those moments when I worked for an ISP/datacenter. I brought down the Internet for an entire county once, accidentally taking up the entire pipe for a few minutes. I also killed DNS for a local, very large hospital for about two minutes, thanks to a misspelled entry in BIND which went live the moment I saved the config file. Good times, good times. (No pressure.)

cisco_trooper

I screwed up PBR during a staged circuit migration once and black holed 250 end users for about 15 seconds once....I made the stupid decision to try using prefix lists rather than access-lists in my PBR definition when I had never used prefix lists for anything else, EVER. Luckily I was monitoring the traffic and immediately knew what had happened.

hiddenknight821

I bricked a customer's home router remotely once when I was trying to upgrade the firmware. Somehow I uploaded the wrong file.

drkat

LOL -

One time our Security department sent me a spreadsheet with computer names that needed to be removed from active directory. I decided to script it.

Well I said "hey lets use powershell" so instead of specifying computer objects I did all objects and needless to say it took out 20 user accounts and mailboxes that needed to be restored and reprovisioned. We used oracle IDM so a lot of approving and re-provisioning lol.. needless to say the boss was... po'd -- fun times

dead_p00l

Forsaken_GA wrote: »

I've brought an entire ISP down before (they neglected to tell me some very important architecture information)

Ive done this before as well as about 50,000 subscribers a different time. Both due to incorrect or missing information in network diagrams.

NightShade1

i brought down a 1000 wireless user network for a few minutes, i was adding a new vlan in a lagg on the wireless controlller, when i was doing a change to that port channel the trusted checkbox damn trusted checkbox i fortgot putting it, its by default in normal ports but not in port channels and i forgot checking it. I loose administration and everything went down, as all the aps does a gre tunnel to the Wireless controller, lucky me i was on site so i just ran to the server room console it and fix it right away.

SteveO86

Issued the wrong debug command on a headend router with 400+ OSPF neighbors... took it down for a few minutes till the router decided to respond again.

Forsaken_GA

dead_p00l wrote: »

Ive done this before as well as about 50,000 subscribers a different time. Both due to incorrect or missing information in network diagrams.

Well, in this case, I was breaking up their networks into vlans, because they had it all chunked together.

They neglected to inform me that their corporate network was in one of those vlans (it was marked differently, but apparently the documentation had been updated, but the work never performed, and everyone had forgotten about it. Doing anything with that network was outside of the scope of my project, so I didn't even look at the configuration for that to make sure the docs were accurate.... lesson learned). As soon as I pull that SVI, everything dies. Everything, including my connection to the router.

Fortunately, I'd been methodical to my approach, saving my work as I went along, and verifying each step. I hauled my ass into the datacenter, consoled in, and rebooted the core router to get back to where the configuration had been just prior, so the total outage was about 15 minutes at 4am... minimal customer impact. But it sure was an ass clenching moment when I saw everything die.

Ended up getting more work out of it. We negotiated a fee to perform the work that needed to be done to make the docs accurate.

dead_p00l

Forsaken_GA wrote: »

Well, in this case, I was breaking up their networks into vlans, because they had it all chunked together.

They neglected to inform me that their corporate network was in one of those vlans (it was marked differently, but apparently the documentation had been updated, but the work never performed, and everyone had forgotten about it. Doing anything with that network was outside of the scope of my project, so I didn't even look at the configuration for that to make sure the docs were accurate.... lesson learned). As soon as I pull that SVI, everything dies. Everything, including my connection to the router.

Fortunately, I'd been methodical to my approach, saving my work as I went along, and verifying each step. I hauled my ass into the datacenter, consoled in, and rebooted the core router to get back to where the configuration had been just prior, so the total outage was about 15 minutes at 4am... minimal customer impact. But it sure was an ass clenching moment when I saw everything die.

Ended up getting more work out of it. We negotiated a fee to perform the work that needed to be done to make the docs accurate.

Fortunately both of mine were during maintenance windows also. The first was similar to yours. It was a market migration from old equipment to new with improperly labeled or documented vlans. About 30 mins of downtime between comparing configs and realizing what happened. The second was a routing engine upgrade(first time i had ever done one). I got the procedure for performing the upgrade from another engineer(who neglected to mention a couple of key points). I ended up with the router coming up partially but not bringing up any interfaces. When I tried to fail over to the second engine it wasn't recognized and immediately failed back to the first. Swapped engines around in the slots with the same result. Turned out to be 1 bad RE. and mismatched code. Ended up rolling back to the previous RE's and got everything back up. Only about 30 mins outside the maintenance window. Learned my lesson to do my own research even if I had instruction from a senior engineer on how to do something.

ChooseLife

Found you! Now, which one of you guys misconfigured my home Internet connection earlier this week and assigned a 192.168.x.x IP to my router's public interface in the middle of the night right when I was fixing a failed storage on a production server? Luckily, I had tethering...

instant000

ChooseLife wrote: »

Found you! Now, which one of you guys misconfigured my home Internet connection earlier this week and assigned a 192.168.x.x IP to my router's public interface in the middle of the night right when I was fixing a failed storage on a production server? Luckily, I had tethering...

*ahem* I call that "Carrier Grade NAT"

Carrier-grade NAT - Wikipedia, the free encyclopedia

Forsaken_GA

instant000 wrote: »

*ahem* I call that "Carrier Grade NAT"

Carrier-grade NAT - Wikipedia, the free encyclopedia

Which I, in turn, call 'Oh, I see you've decided you no longer want my business'

Bl8ckr0uter

Most recently during I did a seamless failover of an ASA from the primary to the secondary (to do work on the former) and I took down our web pages for about 15 minutes. Turned out to be an arp issue (which ironically didn't fix itself when I failed back). Clearing the arp cache fixed the issue.

Forsaken_GA

Bl8ckr0uter wrote: »

Most recently during I did a seamless failover of an ASA from the primary to the secondary (to do work on the former) and I took down our web pages for about 15 minutes. Turned out to be an arp issue (which ironically didn't fix itself when I failed back). Clearing the arp cache fixed the issue.

You will be surprised at how often that's necessary if you're not deploying something like HSRP. When I was still working for a web hosting company, ARP problems were the bane of my existence.

Bl8ckr0uter

Forsaken_GA wrote: »

You will be surprised at how often that's necessary if you're not deploying something like HSRP. When I was still working for a web hosting company, ARP problems were the bane of my existence.

I learned a few more times in the lab Arp is pretty important and it doesn't seem to be that stable.

With stateful failover, it does copy the over the arp table. This particular firewall didn't have stateful failover (although it did have failover). It wasn't that big of a deal since it was late a night but I was knocking my head against the wall and telling my boss that I didn't change any rules and etc. It was pretty funny when it was over.

tpatt100

I remember sending out an email to everyone about "shift changes" and it came out "sh*t changes" I got all kinds of funny replies.

nel

Dude, thats why we have managers. let them send out the correspondence . If not, always get stuff proof read.

swild

Just because of this, I always use Word to type company wide emails. Copy-Paste. One day I will have my own secretary to proof my letters.

hackman2007

Look at the bright side, at least you didn't misspell regards (misspelled as retards) .

I've never done that, but then again, that's why I don't use the word "regards".

Everyone

nel wrote: »

Dude, thats why we have managers. let them send out the correspondence . If not, always get stuff proof read.

Yeah but then you have people like me who sneak funny stuff into company wide e-mails past managers.

I "Rick Rolled" all 3000 employees at the last company I worked for in an e-mail. The e-mail was to announce our migration to Exchange 2010. I sent it to my manager for approval, he approved it and sent it to our PR department, PR department sent it out to the whole company. Lucky for me, even though PR didn't get the joke (I actually had to explain it to them later, and I'm not sure if my manager got it or not) it was a huge hit. The unlucky part was that people liked it so much that they made me the spokesperson for the IT department. From then on I had to come up with clever ways to get the word out on major IT changes.

VAHokie56

While doing a closet upgrade at a hospital once we had our cable dudes go threw and use sheers to cut all the copper out of a 4507 and then killed its uplinks and yanked it out...5 minutes later some angry nurses came out in sheer panic that some device wasn't working and they had a guy In the middle of some sort of operation ( no idea how serious). Just had to throw a little 2950 back in and patch it back real quick but boy oh boy was I scared...not really our fault however, some PM crapped the bed on that.

pham0329

Reading some of these responses makes me feel better about the dumb mistakes that I have made in the past.

cisco_trooper wrote: »

I screwed up PBR during a staged circuit migration once and black holed 250 end users for about 15 seconds once....I made the stupid decision to try using prefix lists rather than access-lists in my PBR definition when I had never used prefix lists for anything else, EVER. Luckily I was monitoring the traffic and immediately knew what had happened.

Can you do PBR with prefix list? I remember trying that in a lab and it didn't work. I think Forsaken or someone else mentioned that prefix list aren't designed for PBR..or something like that.

SteveLord

I find that when I send out emails with instructions, containing steps and even screenshots...I always manage to forget how something may look on someone else's screen. That minor difference in appearance throws at least 1 person off.

LinuxRacr

The company I worked for would hire out another company to clean the data center floor every six months. The cleaning company would have to use sockets to plug in their cleaning equipment. It seems that we would always get some type of outage right about the time the cleaners were there....lol. They would sometimes randomly unplug stuff..

demonfurbie

"ohh look a usb key with my fav football team on the side, i wonder who lost it maybe they have an email or something on it"


normally followed by ....

"why is my computer running slow"

Moki99

This thread is hilarious! I haven't done anything too crazy yet but I know it's coming >.<

Akaricloud

I had a user move desks and was trying to swap their phone over to it. Well they neglected to tell me that the jack IDs are each used twice in the building(Why? -I have no clue).

I ended up swapping a customer service line with a sales line midcall. It still perplexes me why it didn't drop either call but all of a sudden they were talking to the wrong people.

Oops?