Whats the most hilarious misconfiguration you've come across on the job?

Ok, maybe not "hilarious" but rather something you found that was configured by someone else that you couldn't help but laugh about. Yesterday, we lost one of our SCCM distribution points due to a single drive failure and it has been deemed unrecoverable. Why? Because someone decided to build the server with 3 drives in a RAID 0 config... A single drive failed and now the server will not come up. Whats more? The server was never added to the networker backup jobs so we couldn't restore from there either. Now, I will have to rebuild this server this weekend. I really couldn't help but laugh when we discovered this. I thought it was insane to do something like this on a production server. They had even set it up with different volumes... The person who did it is no longer working at our company (They left for another job about a year ago).

Find more posts tagged with

Comments

Essendon

A small 10 day contract job that I did many years ago - a core router connecting supermarket stores to the main office was hanging by its cables behind the rack it was supposed to be in. You'd hope there was a second router providing redundancy, but in a place like that I doubted it!

shodown

I did a network redesign for a health care provider some years ago. The entire internal IP space was made up of public IP's that were all over the place. When I was cuttting over the new network the provisioning engineer for the ISP told me about having to filter all there traffic for this customer as the internal IT guy was a idiot.

demonfurbie

wep wireless used in a lawyers office with no antivirus or firewalls ... because they slowed down his computer

jibbajabba

I was called on an emergency in the middle of the night because their fileserver could not boot any more. It was down from its reboot of Saturday morning until Sunday night when I was called. I went down there, checked the DC, pulled the floppy disk from the drive the server was trying to boot from and went back to bed ....

LittleBIT

AVG, Norton, McAfee, Kaspersky, Panda AV, Webroot, and about 5 toolbars installed. The 5 AV's were, as explained by the user, 'To protect me against virus's. I assumed more was better". 5 Toolbars, Ask Jeeves, Google, Yahoo, and 2 malicious ones.

The original ticket subject line: 'Can't update antivirus, error'.

wikiget

While trying to fix a person's wireless network, I find his router sitting on a shelf with nothing plugged in (including power).

When I ask about it he responds, "I thought wireless means no wires. What a ripoff."

eansdad

demonfurbie wrote: »

wep wireless used in a lawyers office with no antivirus or firewalls ... because they slowed down his computer

Worst part is this is a common thing in small law offices.

Sad thing is almost all of these I've seen.

We had a guy doing a middleware update and didn't reboot the dozen or so servers as the instructions stated during a planned update. Wasn't until after he left that problems started happening. Had to back out the changes and redo them because of a simple reboot.

FloOz

This one is not really a misconfiguration but more an easter egg. If anyone has access to an Arista switch try the command "show donkeys"

NightShade1

I dont have an arista switch:P post the output!

Hypntick

Aside from the horrible cable jobs, which I think we've all seen...
ESXi server set up on a piece of hardware with a software RAID controller that was not supported. They went ahead and installed anyways, no RAID. Glad I didn't get that particular issue.

Fulcrum45

I had an I.T. Manager (at least that was his official title anyway) email me to tell me half of his network was down. Unable to troubleshoot remotely, I went onsite only to discover that he was setting up new workstations just prior to the incident. After isolating the trouble ports- this took a while-I asked to visit the room where he had set up the computers only to discover that it was locked and that they had sent everyone home- to include the people who had the key. The next day I went back and got access to said room. What did I discover? A 6ft network cable coming out of one network drop and back into another. I had to take a picture of it. He didn't fully comprehend what he had done wrong but covered it up well with the powers that be- hence the reason he's still their I.T. Manager.

datacomboss

demonfurbie wrote: »

wep wireless used in a lawyers office with no antivirus or firewalls ... because they slowed down his computer

Seen that too with server, Quickbooks, router and PC admin-level passwords of 12345 and for more security 123456

--chris--

I only have a few months under my belt but what I lack in quantity I make up for in quality.

I have seen a customers entire Avaya server run without any type of RAID/backup on "8 year old hard drives". I cant confirm the age of the HDDs, but can confirm the first half of that.

All employee data (email addresses, windows user names, passwords, proprietary software logins, etc) were stored in an excel file with a password that was...hrmmm...inappropriate. Glad there were not female members of the IT team while I was there.

Same place had a problem with employees storing "team" data (excel documents) locally making them either unavailable if xxxxxx person was gone or if a HDD crapped out completely gone. So they mandated all team data to be stored on one of the many servers.

Problem with this? The "team data" folder became corrupted. No backups. All gone. Some of the team members still had versions of the files stored locally and they were working on rebuilding the massive excel files when I last seen them...

The financial controllers desktop had constant malware problems that were never resolved, but merely band-aid'd into oblivion. The appropriate course of action to fix this? Revert to an earlier date.

it_consultant

NightShade1 wrote: »

I dont have an arista switch:P post the output!

Arista Networks switch easter egg | FireDaemon

eansdad

it_consultant wrote: »

Arista Networks switch easter egg | FireDaemon

LOVE IT...

I got another one....One of our techs calls about one of his buildings internet going down. Find out he had just built a new print server the day before....wait for it...He enabled DHCP on this server...Why you would turn on DHCP on a print server is beyond me. Lucky for him he was well connected so he still has a job. They just move him around year after year sending in a much more qualified tech to clean up his mess.

Had a teacher not once but twice bring in a home wireless router because we didn't have wireless in this building yet. Yup...never disabled DHCP so he took down the building each time....

Not a configuration issue but certainly funny...Assistant Superintendent, who was the IT Supervisor at a different school district and supposedly CCNA certified, said to me..."Why would we need static IPs on these switches?" This was after I complained about not having the switch passwords for a new building I was put in and finding that half the switches were on DHCP.

Yes this place is that bad....

Fulcrum45

eansdad wrote: »

LOVE IT...

I got another one....One of our techs calls about one of his buildings internet going down. Find out he had just built a new print server the day before....wait for it...He enabled DHCP on this server...Why you would turn on DHCP on a print server is beyond me. Lucky for him he was well connected so he still has a job. They just move him around year after year sending in a much more qualified tech to clean up his mess.

Had a teacher not once but twice bring in a home wireless router because we didn't have wireless in this building yet. Yup...never disabled DHCP so he took down the building each time....

Not a configuration issue but certainly funny...Assistant Superintendent, who was the IT Supervisor at a different school district and supposedly CCNA certified, said to me..."Why would we need static IPs on these switches?" This was after I complained about not having the switch passwords for a new building I was put in and finding that half the switches were on DHCP.

Yes this place is that bad....

Ugh- I have a customer who is notorious for that. They have hundreds of employees who work in several building strung together via fiber. Every now and then when "THE INTERNET IS DOWN!!!!" they spaz and I have to find the guy who thought he'd bring in his wireless from home....and leaves DHCP on. Fairly difficult when you have 7 building and only two subnets.

it_consultant

When I was just starting my career I kept learning about "Rogue DHCP servers" and I remember thinking, "how much of a problem could that be?" Well, since everything ever produced has the capability of being a DHCP server, quite a bit of one apparently.

bull313

Do ID-10-T errors count?

jvrlopez

A brand new in box, $10,000 encyrption device used as a door stop for about a year.

datacomboss

datacomboss wrote: »

Seen that too with server, Quickbooks, router and PC admin-level passwords of 12345 and for more security 123456

Apparently, it is the most common password.

Epic Facepalm as "123456" Replaces "Password" as Most Common Password - Reviewed.com Laptops

jahaziel

eansdad wrote: »

LOVE IT...

I got another one....One of our techs calls about one of his buildings internet going down. Find out he had just built a new print server the day before....wait for it...He enabled DHCP on this server...Why you would turn on DHCP on a print server is beyond me. Lucky for him he was well connected so he still has a job. They just move him around year after year sending in a much more qualified tech to clean up his mess.

Had a teacher not once but twice bring in a home wireless router because we didn't have wireless in this building yet. Yup...never disabled DHCP so he took down the building each time....

Not a configuration issue but certainly funny...Assistant Superintendent, who was the IT Supervisor at a different school district and supposedly CCNA certified, said to me..."Why would we need static IPs on these switches?" This was after I complained about not having the switch passwords for a new building I was put in and finding that half the switches were on DHCP.

Yes this place is that bad....

Is your network 192.168.X.X?
If its not I really can't see how it can bring down a network. I would love to see the answer. Curious on how this affects the network.

darkerz

BGPv4 IP prefix list,

...

I have to "Add" to it?

Nah lets replace the entire list of subnets with just the one I'm trying to add.

Syntax & Watch The WORLD BURN fail.

--chris--

jvrlopez wrote: »

A brand new in box, $10,000 encyrption device used as a door stop for about a year.

Didn't consider that type of mistake.

I can say I have seen similar; $3200 media gateway used to keep the supply closet door open while moving boxes in and out. "Its in the box still, its ok".

That reminded me of one I forgot about. Someone signed up for a P&S warranty on some IBM blades (two dozen of them). He thought the contract was $21,000 for 36 months. It was $21,000 for every 12 months (with a 36 month commitment). It was my job to try and get out of it or diminish the cost any way possible before his boss figured it out.

--chris--

jahaziel wrote: »

Is your network 192.168.X.X?
If its not I really can't see how it can bring down a network. I would love to see the answer. Curious on how this affects the network.

This question has me wondering. Since I only have had one real IT job, is it common for enterprise level networks to use the 192.168.x.x mask?

Iristheangel

I don't know if it's necessarily "hilarious" configurations if I have to go clean it up but here are a few I've seen:
-IGMP snooping turned off in a datacenter
-IP CEF turned off of the distribution and core switches in a data center
-Automatic VTP Pruning enabled on some switches and disabled on others on the same LAN and the havoc it created...
-Debugging left on... for years.... without ANYONE noticing.
-Solarwinds sitting on the network yet no alerting set up or devices configured to send logs or trap information to it
-Telnet set up. SSH blocked. On EVERYTHING you would think of as essential
-Devices with local logins that everyone forgot and that no one has been able to log into for years
-Not really a configuration but I saw someone order a T1 line for the wrong address because he didn't like talking to the customers and didn't want to even call them to get the right address.... He ended up quitting and the project got handed to me. It was LOADS of fun telling them they would have to wait 30-45 more days to have network connectivity to their new office
-Redistribution between OSPF and EIGRP that was a godawful mess...
-Wiring vendors who wired a whole office and did it so poorly that the network closets look like spaghetti and the failure rate on each copper run is 20%

I could go on but I'm having one of those days where my head hurts. My job consists of 60% projects and 40% of going back to fix these big mess-ups as I see them

shodown

Iristheangel wrote: »

-Debugging left on... for years.... without ANYONE noticing.

I'm super guilty of this one. I did it today in fact working with someone. I guess I'll get around to shutting it off.

beaucaldwell

last company I worked for, a leading biotech company that does things like revolutionizing DNA sequencing (among other things) refuses to setup backup servers to backup user data of any kind even for their executives... can't begin to tell you the 10's of thousands of dollars I've seen them waste sending HDD's to Kroll for recovery only to get nothing back... to make things better, legal does not allow end-users to use external HDD's to backup their own things...

RouteMyPacket

eansdad wrote: »

LOVE IT...

I got another one....One of our techs calls about one of his buildings internet going down. Find out he had just built a new print server the day before....wait for it...He enabled DHCP on this server...Why you would turn on DHCP on a print server is beyond me. Lucky for him he was well connected so he still has a job. They just move him around year after year sending in a much more qualified tech to clean up his mess.

Had a teacher not once but twice bring in a home wireless router because we didn't have wireless in this building yet. Yup...never disabled DHCP so he took down the building each time....

Not a configuration issue but certainly funny...Assistant Superintendent, who was the IT Supervisor at a different school district and supposedly CCNA certified, said to me..."Why would we need static IPs on these switches?" This was after I complained about not having the switch passwords for a new building I was put in and finding that half the switches were on DHCP.

Yes this place is that bad....

So why didn't you secure the network from rogue DHCP servers?

jahaziel

--chris-- wrote: »

This question has me wondering. Since I only have had one real IT job, is it common for enterprise level networks to use the 192.168.x.x mask?

I believe I only seen it in really small companies. I'm more used to 10.X.X.X being sub-netted to meet your needs.

CodeBlox

jahaziel wrote: »

Is your network 192.168.X.X?
If its not I really can't see how it can bring down a network. I would love to see the answer. Curious on how this affects the network.

If the wireless "router" is connected to an access port on your switch and your PC sends a broadcast DHCP request, the wireless "router" could respond and make an offer before the legitimate DHCP server and you end up with the 192.168.x.x address which likely won't belong to a real subnet in your environment. You'd probably catch a mom and pop shop using it though.