Stupid Things You've Done?

ColbyG

Last week I configured a bunch of ports for our server team. I didn't think to put a "no shut" command in my text file I was pasting. I didn't imagine the ports would be shutdown. Needless to say, the server guys weren't happy, lol...

Anyone else have good mistake stories?

RobertKaucher

I knew a consumer tech that was putting a laptop back together and put a long screw in a spot that took a short screw drilling it through the mobo, which screwed up the mobo (pun intended).

Let's see... I misunderstood the owner of a company and put his iPhone to synch with Exchange deleting 50% of his calendar....

mrmcmint

I accidentally deleted a WSUS group, much to my own annoyance; it was customized with lots of updates other groups didn't get. It took a while to build that back up

blargoe

One time I was really tired from working the night before probably. I had probably 12 Remote Desktop windows open to various servers and needed to reboot a dev server or something that was OK to reboot in the middle of the day. Instead I rebooted the wrong RDP session - our production Exchange mailbox server and took down everyone's email for 20 minutes.

To make matters worse we had a newbie help desk person, who told the first few people that called "Yeah, I know, _______ accidentally rebooted the server" instead of just saying that it was a known issue and it would be back online in a few minutes.

undomiel

Back in my early administrative days I implemented ipsec on the sole DC. Which of course resulted in enhanced security of it being unable to talk with any clients or other servers. Fortunately I caught that really fast and was able to sweep things up before the locals caught on.

GT-Rob

I could probably hit the max characters on a reply with "stupid mistakes". Its part of the job, and as long as you learn from them I guess its important.


Hmm off the top of my head lately.... I advertised a static route into our OSPF domain for pretty much no reason other than it was on the 'old' router, meanwhile breaking the route to that destination.

I separated a NAT entry on our firewall and made sure to update the rule base to allow the internal address, but forgot to do the new external addresses, effectively cutting out our customers into our website for an hour.

One of my guys moved a bunch of cables around on a switch upgrade (going from 24 port to 4, and didn't move the configs with them. About 30 people were in the wrong vlan or had their port turned off on Monday.

Our ISP recently rate limited us to 300k instead of 3000k (their mistake, not ours).

A removed an old router that an even older server was using as its gateway (even though this router was a dialup VPN router...)



Sometimes you dont have control over these things, and sometimes they are 'learning exercises'

phoeneous

I restored an old image to a workstation not knowing it had a static ip already configured. That static ip was identical to a server in production. Mistakingly, I plugged it into a server vlan port. When the workstation booted up, arp tables got screwed up. The server that it was conflicting with was an application server that controlled our general ledger. This was at a bank...

My boss didnt believe me when I told him that I didnt know the image had a static ip configured so he spent a few hours searching eventvwr just so he can bust me.

Pash

Errrrm i have too many to count now lol

Recent one.....I tried the fixmbr fixboot commands from xp recovery console to try and restore the boot order on an safeboot encrypted hdd....yeh that wasnt a smart idea - partition tables were gone after that.

keenon

Late one night I brought up a new bgp peer and was slow about putting the filter in for blocking routes between different AS ( normally i'd have the interface to the peer shut). So my one peer flooded 2 others

UnixGeek

I once installed OpenManage on a rack of servers, and had to update some libraries in the process. I forgot to run the prelink script before signing off, and update the monitoring system's checksums. 12 hours later, the poor guy who had the pager was woken up in when the nighly prelink run was complete, and the checksums didn't match. He thought someone had rooted those boxes.

rsutton

Some good stories here. For me:

Taken down the internet with an incorrect configuration on our ASA.
Rebooted servers by having RDP sessions open that I thought were my local system.
Putting an improperly configured Linux web server on the internet and having it promptly get rooted.
Improperly configured DFS and lost a bunch of data.

There are many, many more, but I think is enough for one thread.

tiersten

Moved /lib by accident. Managed to recover from that however without downing the system.

ColbyG

These are some good ones, hahha.

Paul Boz

Shut down fiber to the home service for 2000 users because I pulled the wrong line card in a Calix aggregator. The Calix rep admitted to telling me to pull the wrong card but it didn't help me any!

Kaminsky

Had a remote site fall over with about 450 user not getting to the NT server. I couldn't remote in so drove 40 miles and switched the server screen on. Duplicate IP lockout. Walked into the pc supprt room where they were busily deploying new PCs. Junior PC guy says "ahh great.. people are telling us the server is in trouble and after that could you help us with these PCs that arn't working when we put them out?" Problem: Duplicate IP.. I ask "what IP address range are you using ?" They say ... "Range ? were just putting this ip address on" .. ME: "what on all of them?" ... "errm yeah..... ?" FFS !!! Me again: "What IP are you using ... ARGGGG! That's the server's IP address !!!!!!" They thought they were pointing it at the dhcp server which they knew was the site server....

There was also the famous story of a guy phoning the helpdesk and saying he spilt coffee in his keyboard and being told that if he washes it under the bathroom sink and leave it next to a radiator to dry for at least 3 days, it should be fine. Later on that day, the helpdesk heard that one of the senior executives had been sacked by the chief exec after he waslked into the bathroom and found the culprit with his laptop under the sink.

My all time favourite was one I heard at college in fortran class. A probe sent into deep space that was programmed to use the gravity of one planet to shoot off to another panet's gravity and then a third and then off into deep space, sending back images for as long as it could. Turned out that it went around the first planet some two years after launch and then went off at a tangent and was never heard off again. After a further two yeasr of analysis of the code, it was all down to the programmer putting in a semi colon instead of a colon.

aordal

I had an xcacls script remove the permissions off the root of the C: drive of a fileserver.

Hyper-Me

They say that an expert is merely someone who has made all possible mistakes

Having said that, I was replacing a backup NIC array in one of our EqualLogic SANs and the tech from equallogic told me to initialize it after I put it in. What he failed to mention was that it would take the primary offline for a few moments while it copied over the configuration files.

About 6000 users thought they lost all of their data in their home folders.

mikedisd2

Accidentally ordered a 110v fuser for a large 220v colour printer. The printer melted.

NightShade03

I write the web applications for our internal web portal. On of the functions that I built in was the ability for HR to look up users in AD and click a "Terminate" button to lockout the account (all with the click of a button from a nice looking web portal). I got a call 20 minutes after it went live that someones account was locked and it turns out then function I wrote didn't terminate properly and got stuck in a loop locking out every single active directory computer & user....brought down the whole network for an hour. Not my best moment.....

the_Grinch

Recently I came in on a Sunday to help put in a battery back-up. The contractor asked if we should move a switch from the floor to the rack (shouldn't have been on the floor I know) so I said sure. The next day one user complained of no internet and a group of users complained that they couldn't print to a network printer. So I talk to a lady in my department and she says go down to the server room (.250 miles away) and restart the spooler service (we can't remote into the machine). I hoof it down, restart the spooler, and come back up. Two hours later I hear that they still can't print. I ping the IP and I get no response. So I go upstairs, check the port number, come back down, and see that I didn't have three ethernet cords plugged into the switch. One was the printer (users rejoiced), another was the lady with no internet, and to this day I don't know what the third one was lol!

tiersten

Kaminsky wrote: »

My all time favourite was one I heard at college in fortran class. A probe sent into deep space that was programmed to use the gravity of one planet to shoot off to another panet's gravity and then a third and then off into deep space, sending back images for as long as it could. Turned out that it went around the first planet some two years after launch and then went off at a tangent and was never heard off again. After a further two yeasr of analysis of the code, it was all down to the programmer putting in a semi colon instead of a colon.

Hmm. Maybe this is a variant of the Mariner 1 story?

There are plenty of space related disasters from mistakes during design or coding.

Mars Climate Orbiter was destroyed because the software that was doing calculations on Earth was giving results in pounds whilst the probe itself wanted newtons.

Ariane 5 rocket exploded because they reused old software with a new trajectory that wasn't compatible with the software. Integer overflow and the primary computer dumped core. This was shortly followed by the backup computer crashing due to the same problem...

eMeS

I used to mostly stay in trouble when I was a direct employee...I worked mostly in a really unhealthy, highly politically charged environment that tended to bring out the worst in people.

1. Calling one of my peers (we were all directs to the CIO) one time a bloated mog. I'm pretty sure she didn't know what "mog" means, but she got the "bloated" part.

2. Before I got married one time I was out on a date on a Saturday night. The young lady I was with wanted to see where I worked so I took her to the corporate campus. Part of our organization at the time included the NOC, and I knew this would impress her so I took her in. Walked in on 2 guys in a compromising position giving each other the good news. They got fired on the spot, I got to man that NOC until some other people could come in. I also got permanent retina burn of that image in my mind.

3. Long time ago, doing a stand alone **** of a mainframe one time, which is basically dumping all of the values in working memory to a tape that can be analyzed. This was a large mainframe environment with several unique machines and several LPARS. Took the stand alone **** on the wrong one...that's a bad thing to do.

4. Drinking that drink dynamik bought me at the bar....

MS

msteinhilber

eMeS wrote: »

Walked in on 2 guys in a compromising position giving each other the good news.

I never knew you used to work with dynamik.

I've done my share of stupid things. Most recently would be when a remote office called for support, they couldn't print nor access the network share and I couldn't remote access the server. Since they were a relatively small office (4-5 users) the server backed up to an external USB hard drive. At some other point it was determined that when the server was initially configured, it was setup to attempt to boot from USB as a result would fail to boot if the external drive was connected. Well I had been coming down with an illness and really wasn't in much of a mood to be working at all. Not thinking straight I said it sounded like it would need to be looked at. Setup a couple of the users to print directly to the printer since it was late on a Friday and somebody would be there Monday to take a look. I ended up being out sick Monday since my illness worsened so my boss had to drive out there (about 8 hours round trip). Turns out it was just a power event or something that otherwise caused the server to reboot and it was attempting to boot from the external drive - whoops.

binarysoul

Got married!

BroadcastStorm

binarysoul wrote: »

Got married!

+ 1 now divorced...

I modified a DC default DC gpo, as a result replication has stopped, and if I remember correctly service account authentication have had issues.

Kasor

Forgot to change the username and password for the intern....!

PetterD

I was out at a customer site and alone in the server-room. I had just finished a successfull upgrade of their Firewall-1 cluster and was quite happy with myself, since everything had went so smoothly.

I had my laptop connected to the switch so i could test traffic while i was upgrading(there were about 500+ switchports in the switch-rack).

I was just going to clean up after myself, and remove the cable i was connected to from the switch when i noticed that although i removed the cable, i was still connected.. I had removed the wrong cable from the switch, and didnt have a clue which port i should connect it to.

After a few minutes the customer came down in to the server room, asking me if there were any problems, because all the employees on his floor (alot of people...) had lost all connectivity. I told him i was still upgrading but that it should soon be up and running (a little lie there..)

Took me about 15 minutes on the phone with the switch-guy who finally found a important port without link to get it up and running again..

SRTMCSE

While scanning/patching servers, I scanned all server subnets (over 250 servers across 4 subnets). Highlighted the servers I wished to patch (only 20 were scheduled), but instead of selecting "Deploy to SELECTED machines", I hit, "Deploy to ALL machines". Needless to say our monitoring software promptly flooded me with hundreds of SNMP alerts, ping failures, service check failures.

I immediately thought I was going to be out of a job, but got myself out of it. The servers were severely behind in patching, and thank God there were no issues the next day (I work nights)...in the end, several of the managers were actually happy b/c it saved us 2-3 months of slowly catching up with patching.

GT-Rob

PetterD wrote: »

I was out at a customer site and alone in the server-room. I had just finished a successfull upgrade of their Firewall-1 cluster and was quite happy with myself, since everything had went so smoothly.

I had my laptop connected to the switch so i could test traffic while i was upgrading(there were about 500+ switchports in the switch-rack).

I was just going to clean up after myself, and remove the cable i was connected to from the switch when i noticed that although i removed the cable, i was still connected.. I had removed the wrong cable from the switch, and didnt have a clue which port i should connect it to.

After a few minutes the customer came down in to the server room, asking me if there were any problems, because all the employees on his floor (alot of people...) had lost all connectivity. I told him i was still upgrading but that it should soon be up and running (a little lie there..)

Took me about 15 minutes on the phone with the switch-guy who finally found a important port without link to get it up and running again..

ugh, that is the WORST feeling, when you have disconnected something from a busy switch stack, and have NO idea where it came from after. Its a stupid mistake but far too easy to make.

disi

Long ago, but I will never forget:
I Got a request from the "IT Manager" to create a security group for them with access to the whole file server...

Group created
connected to the fileserver...

"replace permission entries for all child objects with entries shown here..."

After that I had to add all the other security groups to the other folders again

RobertKaucher

eMeS wrote: »

Walked in on 2 guys in a compromising position giving each other the good news. They got fired on the spot, I got to man that NOC until some other people could come in. I also got permanent retina burn of that image in my mind.

MS

I am eagerly awaiting your book detailing all of the stories you have like this... It would sell well I am sure.