What the most embarrassing technical mistake you ever made?

GOZCU

I was playing with the company's CMS. And after a small mistake web site was down . URL was not working any more. It was 6 pm and there was a meeting with important customers next day at the morning. Aaa, the most critical issue was; there was no backup of the web site. (don't ask me why, it was my first week at the job)


but thanks God, Google cache helped and charmingly it was working again before the meeting.

f0rgiv3n

A couple jobs back I was setting up a GPO for a user that was going to be shared and used on PCs that were publicly accessible. So I created the GPO, and went to town locking down things like removing the C:\ from view, no options menus, manually changed the desktop wallpaper to one I randomly created for test purposes, etc...

I went home.

Well, I was a newbie and forgot that the GPOs have "Authenticated Users" in the apply-to by default and at this company we linked ALL GPOs at the domain level. Soooooooooooo..... Every single person that came in and logged into their computer that morning got that new lock-down policy including the wallpaper. Luckily it wasn't something ridiculous, just like a picture of a mountain with some text on it. There were quite a few people who had to reboot, etc... in order to gain access to their C:\ again


Also, last job, was changing some ports vlans and using up arrow to repopulate the command. "up, backspace, backspace, type port #, enter". Put in the wrong port number just from fat fingering it and hit enter out of habit... Changing the VLAN for the Internet port for the entire place. I sort of freaked out and wasn't thinking right to change it back in a quick moment (i couldn't quickly remember what vlan it belonged to previously) so I hopped over to my coworkers cube "DUDE, i just changed the VLAN for the Internet port..." He was able to get it changed quickly back because he wasn't all freaking out like me .

Another time, I was testing an ASA upgrade, and so upgraded the secondary firewall in an ASA cluster and when the secondary firewall came back up... it took over the primary taking out the entire site until I could get the onsite IT to run over and unplug the newly upgraded firewall.... Oof, that one sucked.

lsud00d

Not a big deal, but doing:

/etc/init.d/process status
/etc/init.d/process restart

And then intending to check the status by using up arrow but accidentally hitting 'restart' again...is a pain when the process takes an extended amount of time to restart

I've gotten into the habit of doing
watch -n 0.5 /etc/init.d/process status
now so I avoid the above problem and can watch it come back after a restart.

4_lom

Dropped a $14,000 server 6 feet onto a concrete floor....

f0rgiv3n

4_lom wrote: »

Dropped a $14,000 server 6 feet onto a concrete floor....

LOL! Did it work after that?

paul78

4_lom wrote: »

Dropped a $14,000 server 6 feet onto a concrete floor....

Yikes.. I hope no one got hurt. Some of the servers these days could crush a foot.

W Stewart

At my previous job fixing linux point of sales systems, I was trying to upgrade flash player for a customer who really should have been using a home PC for whatever they were doing. The system was FC4 and it wouldn't upgrade without upgrading glibc. I wasn't able to find an rpm for glibc and was told that we weren't allowed to compile anything from source so I was just about to give up when a helpful coworker offered me a link to a centos rpm for glibc. It wouldn't take at first so I used some command that I can't remember to force it to install replacing the original version. It caused issues and I stupidly tried to remove the rpm thinking that the fedora rpm might still be on the machine. After that I couldn't even log into the machine so they had to ship the hard drive back to us and I got a talking to. I never heard the end of that and I learned not to mess with anything until I know what I'm doing.

I should also note that I wasn't a complete screw up at that job although I made my fair share of mistakes. I was pretty awesome when it came to figuring out issues with the database on the point of sales and fixed issues that people who had been there for years weren't able to fix. The guy who was probably the most knowledgeable about the point of sales software when I started working there was trying to remember how to fix an issue that he ran into a awhile back. He said that it took him about a week to figure it out but couldn't find the ticket relating to it. Another tech who no longer worked there figured it out in a couple of days but never documented it. I accepted this as a challenged and after looking at the postgres log file I figured it out in 5 minutes while everybody else was still searching through old tickets. I was pretty awesome but they probably would have never paid me what I was worth.

VAHokie56

in the spirit of this thread there is a really cool site with all sorts of funny IT crap ( kinda programming focused) but worth looking at The Daily WTF: Curious Perversions in Information Technology

discount81

A firewall at a remote site which the users required to connect to our data centre, I changed the internal IP while I was working on it, knew EXACTLY what I had done when it dropped the connection.
At the time I had not configured any SSH access as this was a new firewall and I wasn't familiar with it yet.
Had caught the bus to work that day, so I called a taxi instantly to take me to the site, while I am on my way the data centre manager is calling me "Hey your warehouse is down"
"Umm yeah my mistake"

Moral of the story: always turn SSH on before you do any work remotely.

ajs1976

Went to log out of a server and hit restart instead.

Had to restrict access for a terminated employee and reversed the permissions so I locked everyone else out instead.

Trifidw

ajs1976 wrote: »

Went to log out of a server and hit restart instead.

Had to restrict access for a terminated employee and reversed the permissions so I locked everyone else out instead.

I was always paranoid of doing this on Windows 2000 servers. Stupid idea to have to go shutdown to log out on a server.

log32

Well I'm a Sysadmin in an Information Security company, and about 3 weeks after I got started there, I was told to check an old server from year 2000(!) running Fedora Core version 2(!) that serves as DHCP and DNS of the company's R&D LAN environment and see if I could get the data moved to a new server before it explodes together with the rest of the data center
at that moment, I had the urge to ask why me ?? but I said "sure no problem" - no guts no glory I say!
quick note: the server hasn't been booted for a very long time before I got to mess with it probably few years.
I was like..."this can't be good if things get messed up, definitely not as a new sysadmin trying to get himself a good name in a new place"
This is the "Server"




The DHCP was holding IP leases of very important servers and workstations and this had to be saved somehow and moved to another server and can't possibly get lost.
The DNS records simply had to move to the new server which isn't that big of a deal compared to the DHCP task.

I had a good idea as to what is the best way to get this done, replication between the servers: I created a DHCP failover between two Linux servers, the old one would be the primary I thought and a new one would be the secondary, that way, once they "know" of each other and the mode I configured their config files to act as (fail over), they would immediately replicate the dhcp.leases file between them and then I'm all set.

things went wonderful, I kept the two servers replicated for 2 days as a test until the complete shutdown of the old Linux server, but before this happens, I have to configure the new DHCP to act as the only server so it won't search for a secondary (slight config change actually).

and as any DHCP, it broadcasts it's own IP in the network, the gateway IP, and the DNS IP to the requesting clients.
sidenote -> everything is done from the command line - no GUI )
well I was ready, I double checked the last format of the config i'm about to use on my server, things can't possibly go wrong since I already have the lease files and that is the important part, haha yeah..right:
I've made a mistake with the gateway IP, instead of typing 10.10.0.1 -> I typed 10.0.0.1 (habbit from home perhaps?)
this caused all servers to obtain a faulty gateway IP and the R&D department to lose their vSphere connection to the VMWare ESX/ESXis, I found myself fast-walking but not suspiciously to the server's room to connect to each of the ESXs and disable/enable their networking device so they could get the correct gateway again, I think I typed the quickest I ever thought I could that few critical minutes, the down-time was about 5 mins, although no data was actually lost, just the developers connections from vSphere
I was fair enough to say what happened and explain my mistake, I actually think it's minimal compared to what would've happened if that server's AC power supply would explode or something after all these years.

CodeBlox

I had a close encounter. I had two consoles open at work, one for a switch trunked to the core and a new switch. I started clearing the config on the trunked interface and my heart nearly skipped a beat when I thought I was doing it on a very important switch. The reality was that I was making changes on the new switch as expected.

About7Narwhal

Had a VIP come in with his laptop wanting to get it re-imaged. So I went through, did the backup and re-imaged the machine. I replaced his data and was just about good to go. I unhooked the laptop from the wired network and attempted to get a wireless connection and couldn't. I hit the soft key to switch the wireless off then back on and still had nothing. I checked every setting I could think of, re-imaged the device twice, and replaced the wireless card in the asset without luck. After two days of continuous work, I realized that the physical switch for the wireless was still off from when I powered it down during the re-image... Guess it was a case of turning a simple issue into something big.

chmod

cxzar20 wrote: »

Blew away the BGP configuration on a remote device...thank God I had a tech at the site to do a quick power cycle.

God bless the rebbot in command.

olaHalo

myself and a few techs were setting up a rack.
i noticed some poor cable management so i decided to unplug and reroute them.
i knew everything was redundant so it should be fine
the cable i unplugged was for the laptop we were using to configure the SAN and it had no battery.

every now and then ill try to ssh or telnet into a switch/router through putty when im connected via console cable and ill wonder why its not working.

eansdad

I set up a GPO for remote profiles for a school and dropped it under the schools OU in AD. Forgot that the students 600+ log in with a generic account. You ever seen what happens when 100+ of the same profile tries to save at the same time?

I had a friend that was new to linux ... He was cleaning up some old files and used su root rm -rf under the root directory on a production server. Thank god for backups.

jibbajabba

Letter wrote: »

Had two putty sessions open at the same time, one to a core router and one to a test device... you can see where this is going.

Similar to me being conntected to a core configuring vlans while connected to my home cisco 800 configuring an IPSec VPN ... Entered "reload in 10" - guess which one I rebooted ...

Speaking of trunks ... surely everyone who works with Cisco kit forgot at least once the "add" when adding VLANs to an allow list on a trunk port - wiping the whole lot

JasonIT

Speaking of trunks ... surely everyone who works with Cisco kit forgot at least once the "add" when adding VLANs to an allow list on a trunk port - wiping the whole lot [/QUOTE]


I was just thinking about my first time doing this... LOL. Working for an ISP and took down a LOT of customers.

J

DevilWAH

I spent a while today throwing emails backwards and forward to our ISP. They have there own NTP stratum 1 servers backed on to atomic clocks, and peering is restricted to only 2 or 3 IP address from each institute they cover. Last week I decided to change which devices peered to them, and sent in a request to have the new IP's allowed.

So this morning they send back and email telling me its all done. So I add the config on my routers with the NTP addresses... nothing.... send back an email asking them to check, get there engineers involved, I can see my router is sending the packets out, and there engineers are telling me they can see the packets coming in and there NP servers respond....

I sent a few emails insisting that i was not seeing a reply and asking them to check this and that. At some point one of the engineers said. "Can you check you access lists....." Doh!!!

Talk about staring me in the face!!

glenn_33

I forgot to switch out the weekly backup tape with our monthly one once... eeeek, still makes my stomach turn

PurpleIT

I took down the phones in our new office because I crossed the wires for the management and the access vlans (the patch cables are color coded AND labeled) and instead of checking to make sure the correct vlans were configured for the ports I went through 30 minutes of unnecessary troubleshooting. If I had simply looked at the port config and looked at the cables I would have had it fixed in seconds.

bermovick

Was helping one of our techs who was trying to get wake-on-lan magic packets to work across vlans. He was reading a forum thread regarding the issue that said to turn on directed broadcasts and have an access list that allows udp packets of a certain type (I forget which now). Enabling directed broadcast wasn't an issue, but forgetting the implicit deny any at the end of the ACL (that was only allowing those UDP packets) before I applied it to the vlan interface for our management+server vlan was not.

(and after an hour of trying to sign out a laptop from our government people to console in and remove the ACL, I went the nuclear option and turned both power supplies on the core 4500 switch off & back on)

About7Narwhal

Being new to linux, I have very little knowledge of the commands and context. Anyway, I accessed a live server via telnet and was prompted with a "Password will expire in X days." warning. No big deal, ran the passwd command and changed the password without issues. I then elevated to SU and got the same prompt.. and again used the passwd command to change the password. Anyone with linux experience will know exactly what happened here: Because I did not specify my user account while in SU, I changed the root account password.. on a live, backbone system. Needless to say, there was some downtime while they fixed the issue. Live and learn I suppose.

paul78

DevilWAH wrote: »

Talk about staring me in the face!!

LOL - how embarrassing. Doesn't that make you wonder what the ISP support folks are saying about you

jibbajabba

About7Narwhal wrote: »

I changed the root account password..

When I was working in a hosting company the new guys usually done that within their first week of trying to reset customer's password lol ...

bgold87

I was working desktop support a few years ago and managment tasked the intern with cleaning up the equipment room and asked that I watch over him to make sure he doesn't throw anything good away. So while we are gathering everything I spot this old scanner in the corner that's been sitting there for years and gave it a chuck into the dumpster. 1800GOTJUNK came and picked it all up. Apparently, that was a client's $12,000 scanner. We were able to recover the pieces from 1800GOTJUNK and I think the spent another $3,000 repairing it. I admitted to it, and they said if it was so valuable it shouldn't have been left in a corner lol My heart sank for awhile though.

Sett

chmod wrote: »

God bless the rebbot in command.

Screw that, "commit confirmed" ftw !

DevilWAH

paul78 wrote: »

LOL - how embarrassing. Doesn't that make you wonder what the ISP support folks are saying about you

I know most of them by name, it's an academic network, so we have a lot of input in how it develops. Indeed I am going to there conference next month and will have to face them.
I am sure we will laugh about it over dinner, just as long as they don't tell my manager or I will never hear then end of it.

LarryDaMan

In my Army days, I was a Satellite Engineer. At the end of each month right before midnight, we had to re-load new encryption keys. We had two satellite links, each with different keys. At midnight, the satellite links went down. I was sure I had loaded the right keys, but decided to re-load to make sure. My co-worker watched as I grabbed the loading device and re-loaded the crypto. Still down. Obviously it wasn't the cryptography. After working an overnight shift and then staying 6 more hours to troubleshoot, I went home. Like an epiphany while in the car, it popped into my head that I had confused system ID numbers and had loaded the wrong crypto...twice. I discreetly called a friendly co-worker at the site and asked him to re-load and he got to be the hero and thankfully he made-up an alternate fix to spare me some embarrassment.