Stupid Things You've Done?

ColbyG

GT-Rob wrote: »

ugh, that is the WORST feeling, when you have disconnected something from a busy switch stack, and have NO idea where it came from after. Its a stupid mistake but far too easy to make.

Yea, I've done that before too, when I didn't really know much about networking. It took 3 hours to figure out where the cable was supposed to go.

Now I'd just log into the switch, do sh logg (or another vendor's equivalent) and figure out which port just went down.

PetterD

After some years in the field, i have felt it a few times... Its a cold/warm panic-type of feeling;d

The last time i had this feeling, i was installing new firewalls at a company that ran Firewall-1 4.1 on NT4 with StoneBeat HA (shared mac).

I installed a separate solution using ClusterXL (shared IP) with different IP-adresses. Since this was different IP`s and different MAC`s we shouldnt have any problems.. right .. ?

That was until we applied the policy (which had a object using static NAT, that made the new firewalls ARP the current NT4 Firewalls external IP, causing an ip-conflict on the WAN-interface).

I had checked all the hundreds of objects before installing the policy, but overlooked that stupid little node-host object..

Had to plug out all the cables and boot the NT4 to clear out the ip-conflict.

bertieb

- Believing the electrician when he says 'sure, that circuit is off' and then, amidst a sea of sparks, tripping several fuses and soiling myself when pulling out really old wiring in an office
- Accidentally dropping a server rack mount in the cab, watching it fall through the hole in the base and see it cut through a core fibre cable that ran under the cabinet (thankfully a redundant link)
- Removing the wrong card from a live Cisco 7206 and promptly 'disconnecting' several leased line customers for a short while (about the time it takes for a 1st line engineer to leg it from his desk to the comms room and ask 'are you doing any work?')

and so many more, usually in the early hours when sleep deprived and not thinking properly. It must be said though I seem to have achieved a far lower 'numpty' rating over the last few years, I guess you do learn from your mistakes!

PetterD

A friend of mine made a little programming typo when he worked for a company that provided SMS-HTTP connections to different companies. They recieve a SMS-text message using HTTP from their customers, and relay the messages to the recipients operator.
The operator charges my friends company, and they add a fee and send a bill to their customers.

The only problem was that my friend (who were programming the API) made some type of typo in the API creating a loop (which in turn sent out hundreds of thousands more SMS-text messages than they were supposed to during a weekend..) His boss ended upp having to pay 40K USD for the messages that they couldnt charge their customers for..

PetterD

A while ago a large company that i know of had a power outage on all of their servers.

All the servers had dual power supplies, and the customer had an old UPS. When the UPS failed, all the servers lost the power... They had plugged both of the power-circuits from all the racks to the UPS that failed...

networker050184

Latest one, I was making a sub interface on a router for testing and when I was all finished up I was deleting everything. I copied and pasted the sub interface (the wrong sub int) for a no interface xxx and as soon as I hit enter I saw the OSPF neighbors go down and my session went down. I was able to log back in through an adjacent router and paste the sub int back in. Took down a couple hundred customers but fortunately it was only like two minutes and no one noticed.

Mishra

I keep calling the network cable a NIC.... Network Interface Card, Network Interface Cable... SAME THING. It makes sense in my head when I say it.

mrmcmint

"An expert is a man who has made all the mistakes which can be made."

I like it

bellhead

I didn't do it but a tech in my group "she was a legacy employee" when I worked for the phone company as a central office tech overwrote the billing data tapes they were the long distance toll data back in the mid 90's. Each co had a tape along with a case and it was clearly marked on it what office. Well she somehow screwed that one up and overwrote one office's with another office's data. The mistake cost the company around an estimated $50k, and what happened to her? Well after that she was no longer allowed to do anything with any equipment only allowed to wire up pots lines on the frame. No time off or lost pay. People like her give Unions a bad name.

eMeS

RobertKaucher wrote: »

I am eagerly awaiting your book detailing all of the stories you have like this... It would sell well I am sure.

Nothing good comes from going in to work on a Saturday night....nothing!

I guess they had an alert and they got to the line in the OPDOC where it instructed them to blow their nearest same *** coworker or something....

I'm just glad I had a witness with me....

MS

L0gicB0mb508

A coworker and I went to switch out a battery on a redundant UPS and found out they weren't so redundant. This was on a critical military network and some high ranking people were in the middle of an online conference. Bad bad bad day.

Watched the same coworker copy instead of move the transactions logs on an Exchange server that had a full disk. It then filled up another array and took the Email down for about 2 hours. This one again on a critical military network.

Watched our firewall go down due to someone incorrectly setting up the backups running on the config. I was of course the only systems admins that had any clue how to work on it. Took down the entire network. This was a slightly less military critical network though.

Had a lot of problems with a site to site VPN once. I had typo'd something in the config. I couldn't getting the two sites to link up. It took down an energy company for about 4 hours. I finally had someone on the other end go through the main VPN config and then I spotted it. I felt like an a$$ for that one.

I'm sure I've made plenty more mistakes, but those just stick out in my mind.

azjag

I configured an antivirus software on a SBS 2003 server to delete infections instead of clean or quaranteen. Don't know what infected the machine but the AV deleted several files, including system files. Spent the next 20 hours rebuilding the server and restoring from backups.

shodown

Some good ones here

I got too many to mention.

1. Changed a access list without doing a reload in 15 command. Removed all the statements leaving the deny at the end without removing it from a interface. This was a small bakery so no harm done.

2. Very tired after a 12 hour shift and was logged into a call center of a large global credit company. I closed the PG gateway and a few other ICM servers when I just wanted to minamize them. They failed over right away and nobody even missed a beat. I spent the next few days trying to figure out how i was going to fail them back over with nobody noticing.

3. When I 1st started my last job I was givin a task to RMA and replace a WAAS at a customers site after talking to a internal Senior engineer I was told that it would be no problem replacing the WAAS during production hours. WRONG IP connectivity as far as web and email was fine, but Calls coming in and there connection to our hosted ICM enviorment dropped for about 4 min. He took the heat for giving me the bad info, but I got a lot of dirty faces until then.

5. Man I could go on for a while working support at a service provider. Doing internal IT I haven't had too many problems (KNOCK ON WOOD)

CSCOnoob

Shut down the interface since it was taking massive packet loss thinking that it had VSAT as a backup. After several seconds of waiting, I found out that the site didn't have any backup at all. No one is there so I couldn't reboot the router. 1st shift team had to wait for the next day for the router to be rebooted.

Had a annoying field tech that didn't want to check the wiring so I took out the spanning-tree bpduguard enable from the configuration. *BAM* Guess what happened? Good thing it wasn't that bad. I was able to shut the port down in time.

I like to post the stupidest thing I have ever heard in my entire life. I heard this from one IT guy, "Where do you connect the wireless PC on the Cisco switch?" I was like, WTF? How did you get hired?

jojopramos

I still remember the carelessness that I did when I am still a tech support in the late 90's, I mistakenly reformated a member server that has an important application database on it. To make things worst, the backup cannot be restored and there is a huge data on it. It cost me my job, and until now, I always see to it that, that bad experience would be my last on carelessness. Im' happy that until now, I don't do any crazy big IT mistakes (the worst is just mistakenly restarting an exchange server through RDP - 20 mins interruption) because of that experience.

SWM

This ones not really my fault, but I got blamed for it by the customer until it was sorted and I explained the issue.

SBS2003 running on a cheap server (well a pc really) with a no frills $10 SATA raid card (with two mirrored drives containing the cust D partition and all exchange and company data). We just took over the site and our tech noticed read/write errors and bad sector messages in the event logs whilst checking out the server. He opened the cheap raid card software and it indicated drive 1 had bad sectors and was failing, it even provided the drive serial number. Note no other messages or errors in the logs or raid card console.

We organised a new sata hard drive and as the tech could not make it back, I got the job. Organised for 4pm server shutdown, told the cust server down for 20-40 minutes and would then be a little slow whilst the server rebuild the mirror over night. Checked the backed logs and media and noted two weeks worth of backups available.

Shutdown server, identified hard drive SN# and replaced it and restarted. Server booted SLOWLY on existing SCSI C partition. By this time I had the classic "how much longer from numerous managers" and my answer was not much longer, its slow due to a RAID rebuild. Server eventually logged on and I opened up explorer and NO D PARTITION, hence no DATA or Exchange !!!! oh #$%^ No errors in logs just missing D partition. Rebooted server and checked raid console and it would not rebuild and only showed one SATA drive, my new blank one.

Turned out the RAID card was faulty and was no longer mirroring the drives. The RAID functions had failed 2 months previous and the business had been running on one drive only and server had not been shutdown for 2 months. This single drive then developed bad sectors and was the drive I removed !!!

Not knowing about the lack of mirroring and just suspecting I had a faulty raid card , I connected the remaining original hard drive to a plain SATA controller and rebooted and could now see the D partiton and all looked OK. Users all logged on and seemed happy. I left to find a replacement RAID card. It was only after a few hours that we discovered that this drive was 2 months behind in data and users could not find recent files!!!!

Problem now was that users had started storing files and the exchange server had started receiving emails and storing them into the 2 month old drive... aahhhhg

I guess my only stupid mistake was ASSUMING the NTbackup files on the backup media (2 weeks worth) were OK, unfortunatley they were all unreadable.... I should have opened them and verified them first.

So customer had no readable backups, faulty raid card and bad sectors on only drive with valid data!

Getdata back on the bad sector hard drive and a Exchange recovery store operation over a sleepless weekend and all was OK by Monday/ Tuesday !! Oh and a NEW server with decent RAID and drives

SephStorm

User's computer needed a new NIC, when out an bought one, brought it back and went to install it only to realize the plate on the front was too big for the case. Being my entrepreneurial self, i removed the plate, being that the hour was late I closed up the case, only to realize the card came with a plate for smaller cases. Next time I came to the house, someone had installed it

Forsaken_GA

PetterD wrote: »

After a few minutes the customer came down in to the server room, asking me if there were any problems, because all the employees on his floor (alot of people...) had lost all connectivity. I told him i was still upgrading but that it should soon be up and running (a little lie there..)

Edit: On second thought, I don't think I want to relate that particular story on a public forum hehe

mikedisd2

Thought I'd update the AMM for our BladeCentre console. All the fans on the chassis sped up to 100% and the UPS' beeped from overdraw. People asked if the servers should be that noisy because it sounded like a jet taking off. Lasted for 10 minutes.

I told an IBM tech about it later on and he said all the symptoms were correct. Wish I had known that before I did it though. May have saved some gray hair.

mikedisd2

eMeS wrote: »

Calling one of my peers (we were all directs to the CIO) one time a bloated mog. I'm pretty sure she didn't know what "mog" means, but she got the "bloated" part.

OK, so what's a mog, anyone?

djhss68

Forsaken_GA wrote: »

Edit: On second thought, I don't think I want to relate that particular story on a public forum hehe

Too late. I already saw it.

ColbyG

I saw it too.:D

stupidboy

I am sure that anyone that has worked with Dell 1955 Blades will have power cycled the entire chassis, that's 10 blades, instead of the single blade they intended when using the DRAC (that's what I tell me self anyway).

On time, I cursored up to far and fast via a HP iLO. I was supposed to have restarted the SNMP Daemon but had hit the return key and after the screen had updated I had run shutdown -r now .... never good on a backed Oracle server in the middle of a customers live auction

LIVE .... LEARN ... SIMPLE!

dstock7337

A couple dumb-dumb moments of mine:

While deleting an unneeded service from several branch servers (as I had been directed), I accidentally deleted a database service from one of them (containing all of the days transactions). Fortunately enough, we were able to restore the database with the transaction logs and the last backup.

Had been cleaning up active directory for a particular domain for about a couple of months. One day, I had decided to go take my break and wanted to lock my workstation before taking off. Well, I typically have a habit of hitting ctrl+alt+delete then hitting spacebar to select the "lock this computer" option, since the focus is already on it. In this particular instance, I had missed either the ctrl or the alt key, had hit the delete key on the OU containing all of the computer account objects for all of the servers, and hit spacebar to confirm.

As it sounds, the permissions had not been set to prevent such accidents from taking place. I quickly ran to the server room to unplug the backup DC but it was too late as it had already replicated. Was able to perform an authoritative restore to bring it back, while the vice president was hanging out in the server room.

Oh what fun.

eMeS

mikedisd2 wrote: »

OK, so what's a mog, anyone?

twat

another line

stupidboy

dstock7337 wrote: »

Had been cleaning up active directory for a particular domain for about a couple of months. One day, I had decided to go take my break and wanted to lock my workstation before taking off. Well, I typically have a habit of hitting ctrl+alt+delete then hitting spacebar to select the "lock this computer" option, since the focus is already on it. In this particular instance, I had missed either the ctrl or the alt key, had hit the delete key on the OU containing all of the computer account objects for all of the servers, and hit spacebar to confirm./QUOTE]

Made a couple of accidental deletions this way, nothing that serious lucky. I always use WinKey + L it is a safer

RobertKaucher

dstock7337 wrote: »

A couple dumb-dumb moments of mine:

While deleting an unneeded service from several branch servers (as I had been directed), I accidentally deleted a database service from one of them (containing all of the days transactions). Fortunately enough, we were able to restore the database with the transaction logs and the last backup.

Had been cleaning up active directory for a particular domain for about a couple of months. One day, I had decided to go take my break and wanted to lock my workstation before taking off. Well, I typically have a habit of hitting ctrl+alt+delete then hitting spacebar to select the "lock this computer" option, since the focus is already on it. In this particular instance, I had missed either the ctrl or the alt key, had hit the delete key on the OU containing all of the computer account objects for all of the servers, and hit spacebar to confirm.

As it sounds, the permissions had not been set to prevent such accidents from taking place. I quickly ran to the server room to unplug the backup DC but it was too late as it had already replicated. Was able to perform an authoritative restore to bring it back, while the vice president was hanging out in the server room.

Oh what fun.

That sucks! "Windows Key + L".

I can only imagine the sinking feeling... Was the VP aware of what was going on?

ColbyG

dstock7337 wrote: »

A couple dumb-dumb moments of mine:

While deleting an unneeded service from several branch servers (as I had been directed), I accidentally deleted a database service from one of them (containing all of the days transactions). Fortunately enough, we were able to restore the database with the transaction logs and the last backup.

Had been cleaning up active directory for a particular domain for about a couple of months. One day, I had decided to go take my break and wanted to lock my workstation before taking off. Well, I typically have a habit of hitting ctrl+alt+delete then hitting spacebar to select the "lock this computer" option, since the focus is already on it. In this particular instance, I had missed either the ctrl or the alt key, had hit the delete key on the OU containing all of the computer account objects for all of the servers, and hit spacebar to confirm.

As it sounds, the permissions had not been set to prevent such accidents from taking place. I quickly ran to the server room to unplug the backup DC but it was too late as it had already replicated. Was able to perform an authoritative restore to bring it back, while the vice president was hanging out in the server room.

Oh what fun.

Whoa.

blargoe

dstock7337 wrote: »

Had been cleaning up active directory for a particular domain for about a couple of months. One day, I had decided to go take my break and wanted to lock my workstation before taking off. Well, I typically have a habit of hitting ctrl+alt+delete then hitting spacebar to select the "lock this computer" option, since the focus is already on it. In this particular instance, I had missed either the ctrl or the alt key, had hit the delete key on the OU containing all of the computer account objects for all of the servers, and hit spacebar to confirm.

As it sounds, the permissions had not been set to prevent such accidents from taking place. I quickly ran to the server room to unplug the backup DC but it was too late as it had already replicated. Was able to perform an authoritative restore to bring it back, while the vice president was hanging out in the server room.

Oh what fun.

Nice

coax31

I ISCI mapped a backup server to the same SAN volume ISCI mapped to the mail server which contained the Exchange 2003 store. I was new at ISCI and thought I came up with a slick way to backup the store. I corrupted the Exchange store and spent 3 days with MS support bringing it back on line. I was sure they were going to fire me but my director deflected the situation for me. All it comes down to is we all make mistakes just be honest when you screw up the quicker you tell someone you screwed up the faster it can be fixed.