Could a MAC Address flood attack cause IP Conflicts?

JohnnyBiggles

I've never seen a MAC address flood attack so I don't know the exact repercussions of one, but I'm familiar with what it theoretically does, overwhelming a switch. We've had a number of issues the past few days where many of our servers on a single subnet across multiple switches have had IP conflicts and some even couldn't ping to anything, even on the same switch. We thought it may have been everything from bad cables, bad uplinks, server issues, a bad switch even, but not much made any sense. Somehow, things magically came back together and there are no conflicts after we did some movement of cables between switches, but now things are pretty much back to the way they were (physically) so I'm lost as to what could've happened. All this started when no one was around on a Saturday. Was it a Domain Controller issue? Switching issue? Ports seemed to check out ok and all the servers have statically assigned addresses. No viruses have been found on key servers. Could a corrupt domain group policy cause something like this? Mac flood attack? Any assistance would be great. Thanks.

it_consultant

Short answer is no, I would be kind of surprised if this were the case, if these things were all DHCP, maybe because the initial lease could get screwed up by the downed switches - however, if your CAM tables were truly full then ip conflicts would be the least of your problems. It actually sounds like a rogue DHCP server handing out bad addresses to DHCP devices, addresses which are in the same scope as your statically assigned addresses - when Windows detects this it will block traffic for you, which is probably why your servers couldn't ping anything.

I did this to myself when I installed a DHCP server on my laptop to test something and then I forgot to turn off the service.

JohnnyBiggles

Is there any way to detect a rogue DHCP server?

it_consultant

Check this out:

Detect rogue DHCP servers on a network | Symantec Connect Community

JohnnyBiggles

Thanks for the info. I'll check it out.

JohnnyBiggles

So I DL'd that program and I'm seeing a public address that registers somewhere in Japan with an IP lookup, but when I check my firewall I don't see any traffic traversing through it with that address and don't understand how an external IP could be used as an internal DHCP. How can I track this down? It seems to be live because every few minutes it appears and the offered address has changed. I can't ping it either, obviously. Any suggestions?

PurpleIT

Don't get thrown off by the IP of the rogue device, it really isn't a factor. I can stick any IP I want on a device and since DHCP is listening for the broadcasts it doesn't have to be on the same subnet as the rest of the VLAN, as a matter of fact, if I were trying to hide it that's exactly what I would do.

I would start hunting for it by MAC; I haven't used the tool it_consultant mentioned in a long time, so I can't guarantee this would work, but it should...

You should be able to look at your ARP table to get the MAC of this rogue DHCP server.

Look at your MAC tables on your switches and you can trace it back to the correct switch and then finally the exact port it is connected to. Disabling that port should stop the problem, but I would prefer to have physical possession of whatever is attached to that port so I can see if this was malicious or perhaps just someone putting a home router on the network so they can get WiFi for their cell phone.

pamccabe

You might also want to look into DHCP snooping to ensure this doesn't happen in the future. Please update this thread with what you find. It is very interesting.

phoeneous

JohnnyBiggles wrote: »

How can I track this down?

C'mon now, a ccnp should know this!

it_consultant

JohnnyBiggles wrote: »

So I DL'd that program and I'm seeing a public address that registers somewhere in Japan with an IP lookup, but when I check my firewall I don't see any traffic traversing through it with that address and don't understand how an external IP could be used as an internal DHCP. How can I track this down? It seems to be live because every few minutes it appears and the offered address has changed. I can't ping it either, obviously. Any suggestions?

DHCP happens below layer 3 so like PurpleIT mentioned, it doesn't really matter. I wouldn't bother looking on a different subnet though, in order for DHCP to work across routers the routers need to be configured with a DHCP helper address and I think it is a solid bet that you guys didn't put in a DHCP helper for some public IP.

A possible way to address this is to write an ACL blocking all access to that IP and apply it on the ingress and egress of all the switchports in that VLAN.

Trifidw

it_consultant wrote: »

A possible way to address this is to write an ACL blocking all access to that IP and apply it on the ingress and egress of all the switchports in that VLAN.

That's a very roundabout way of doing it...

Laptop > Wireshark > plug into network > get IP address > find the exchange of packets in wireshark capture > find mac address > show mac-address on switch follow it until you find the infected computer > shutdown switchport to it and tell whoever looks after it to sort it out before it is allowed back on the network.

JohnnyBiggles

PurpleIT wrote: »

Don't get thrown off by the IP of the rogue device, it really isn't a factor. I can stick any IP I want on a device and since DHCP is listening for the broadcasts it doesn't have to be on the same subnet as the rest of the VLAN, as a matter of fact, if I were trying to hide it that's exactly what I would do.

I would start hunting for it by MAC; I haven't used the tool it_consultant mentioned in a long time, so I can't guarantee this would work, but it should...

You should be able to look at your ARP table to get the MAC of this rogue DHCP server.

Look at your MAC tables on your switches and you can trace it back to the correct switch and then finally the exact port it is connected to. Disabling that port should stop the problem, but I would prefer to have physical possession of whatever is attached to that port so I can see if this was malicious or perhaps just someone putting a home router on the network so they can get WiFi for their cell phone.

Ok, so what I did was, I ran Wireshark on the PC that was running that DHCP program. When I refreshed it (the DHCP app), the same info came up with that 220.x.x.x address as the DHCP server which was offering an address we have defined in our scope... let's say it's 192.168.100.x. At the same time, I monitored wireshark, and as expected, a Discover packet went out sourced from this machine and an Offer packet came back from 2 of our DHCP servers. I checked the data for both lines that popped up (I have the filter set to "bootp.option.type == 53" to view DHCP traffic only) and neither of the MAC addresses that were reporting IP conflicts or the 220.x.x.x IP were displayed, only the DHCP server on our network. I did this on another suubnet we have that gets DHCP relayed from that same server, and got the same, but the program is NOT showing that 220.x.x.x address, only our DHCP server on 192.168.100.x. So, there IS a difference between the subnets but I can't figure out where this address is coming from and arp checks don't show anything for it either. Now the hard part of all this, is that all this strange activity is on a network that is using "dumb" (unmanaged) Dell switches, so I can't really even log in to check MAC tables and such and, for now, am limited to the devices connected to them. I'm still working on this but if you have any other insight on this that would be great.

phoeneous wrote: »

C'mon now, a ccnp should know this!

I know, I know but this is easier said than done. First off, I just got it a few months ago and while I have it, that is Cisco and almost our entire network is Dell equipment (actually not far from it config-wise, but still)... and as mentioned above, these are dumb switches which predate my role here. Also, we had some other issues occur at the same time so trying to pinpoint things which may or may not be related while trying to tackle basic operations/functionality/availability has been a challenge.

PurpleIT

JohnnyBiggles wrote: »

Ok, so what I did was, I ran Wireshark on the PC that was running that DHCP program. When I refreshed it (the DHCP app), the same info came up with that 220.x.x.x address as the DHCP server which was offering an address we have defined in our scope... let's say it's 192.168.100.x. At the same time, I monitored wireshark, and as expected, a Discover packet went out sourced from this machine and an Offer packet came back from 2 of our DHCP servers. I checked the data for both lines that popped up (I have the filter set to "bootp.option.type == 53" to view DHCP traffic only) and neither of the MAC addresses that were reporting IP conflicts or the 220.x.x.x IP were displayed, only the DHCP server on our network. I did this on another suubnet we have that gets DHCP relayed from that same server, and got the same, but the program is NOT showing that 220.x.x.x address, only our DHCP server on 192.168.100.x. So, there IS a difference between the subnets but I can't figure out where this address is coming from and arp checks don't show anything for it either. Now the hard part of all this, is that all this strange activity is on a network that is using "dumb" (unmanaged) Dell switches, so I can't really even log in to check MAC tables and such and, for now, am limited to the devices connected to them. I'm still working on this but if you have any other insight on this that would be great.

OK, first things first - you got TWO offer packets? Since I don't want to assume anything, have you checked that the scopes don't overlap?

They might not have overlapped a few days ago, but obviously something changed...

I know, I know but this is easier said than done. First off, I just got it a few months ago and while I have it, that is Cisco and almost our entire network is Dell equipment (actually not far from it config-wise, but still)... and as mentioned above, these are dumb switches which predate my role here. Also, we had some other issues occur at the same time so trying to pinpoint things which may or may not be related while trying to tackle basic operations/functionality/availability has been a challenge.

It's not exactly a high-tech way of doing things, but IF you could ping the address you could essentially ping-flood it and look for the constantly blinking light on the switches. Of course that depends on your network traffic, access to switches, etc. You could also start disconnecting switches so you can isolate which switch the rogue is connected to and then start checking the ports one at a time.

Obviously, this would be best done after hours or during a maintenance window, but I am just brain-storming here.

it_consultant

You can also trace each cable. You know whatever device is the problem is plugged into your stack somewhere. Identify where each port goes and verify they are legitimate. I realize that is an awful lot of work but if you bring a labeler, you will be better off for it in the long run.

JohnnyBiggles

PurpleIT wrote: »

OK, first things first - you got TWO offer packets? Since I don't want to assume anything, have you checked that the scopes don't overlap?

They might not have overlapped a few days ago, but obviously something changed...

PurpleIT wrote: »

It's not exactly a high-tech way of doing things, but IF you could ping the address you could essentially ping-flood it and look for the constantly blinking light on the switches. Of course that depends on your network traffic, access to switches, etc. You could also start disconnecting switches so you can isolate which switch the rogue is connected to and then start checking the ports one at a time.

Obviously, this would be best done after hours or during a maintenance window, but I am just brain-storming here.

it_consultant wrote: »

You can also trace each cable. You know whatever device is the problem is plugged into your stack somewhere. Identify where each port goes and verify they are legitimate. I realize that is an awful lot of work but if you bring a labeler, you will be better off for it in the long run.

This is going to be very difficult to do... even after hours...

it_consultant

Well, its a problem that you have production servers connected to dumb switches. It really limits what you can do so spending time to verify all the endpoints becomes a huge part of the job. As others have pointed out, if these were managed you could mirror all the ports and wireshark the monitor port - which would speed up the process considerably. Doing that would reveal the MAC of the offending device, with that you would be able to drill down which switchport that MAC is connected too.

JohnnyBiggles

Before the holidays, I had started further segmenting our network and part of that project was to get thins off those switches so we could update firmware and manage them, so it's already in the works. We just happened to hit a snag with this situation but since everything seems to be running normally now, I'm going to keep a close eye on things until I can get to those switches to reclaim them.

apr911

Hi Johnny,

Admittedly, I havent read through the entire posting here so I dont know what all you have or have not done or found so far but I wanted to point out that it is possible that the MAC address flood may cause erroneous reports of duplicate IPs.

Windows uses ARP to determine if an IP is in use. If an ARP request receives a positive response or the address already exists in the local ARP cache, windows will flag that IP as in use elsewhere on the network and not allow it to be used on that machine. I believe Linux does the same thing.

So yes, a mac address flood can result in the appearance of duplicate IP addresses even when they dont really exist.

it_consultant

apr911 wrote: »

Hi Johnny,

Admittedly, I havent read through the entire posting here so I dont know what all you have or have not done or found so far but I wanted to point out that it is possible that the MAC address flood may cause erroneous reports of duplicate IPs.

Windows uses ARP to determine if an IP is in use. If an ARP request receives a positive response or the address already exists in the local ARP cache, windows will flag that IP as in use elsewhere on the network and not allow it to be used on that machine. I believe Linux does the same thing.

So yes, a mac address flood can result in the appearance of duplicate IP addresses even when they dont really exist.

I would have to see this in a lab in order to believe it. The affect of a flood would be much different than the OP describes, machines would be going offline with NO error or duplicate IPs regularly. As he describes, the go offline AND they alert for a duplicate IP. In the case of a MAC flood attack we would see things go offline in mass quantities without the correlation of the error message in Windows.

f0rgiv3n

Since you're seeing a public IP address show up in your DHCP responses maybe you should look first at any ISP equipment you have onsite. Maybe you have a DSL modem or something similar plugged into the dumb switches? I would check all of your equipment first before assuming there's a malicious user/device. Most of the time it's just an accidental thing, I would go to the closet and just check all of the devices plugged in that exist in the closets.

Also as far as the mac address flood: I think that a mac flood wouldn't cause anything to necessarily respond to an ARP request, nor would it actually cause an outage. So even if that was going on, theoretically it wouldn't be causing this particular problem. Remember, MAC flooding is used to turn a switch into a hub so they can listen to traffic, not necessarily a DoS. Traffic would still flow, it's just that the switch would send all traffic out all ports.

You might try to ping the IP address that you are trying to find in order to fill your arp table and get the MAC address of the remote device.

apr911

it_consultant wrote: »

I would have to see this in a lab in order to believe it. The affect of a flood would be much different than the OP describes, machines would be going offline with NO error or duplicate IPs regularly. As he describes, the go offline AND they alert for a duplicate IP. In the case of a MAC flood attack we would see things go offline in mass quantities without the correlation of the error message in Windows.

True. I guess ultimately it depends on the exact vector being used to cause the MAC flood and what other variables or things are occurring. A MAC flood alone probably would not cause devices to report duplicate IPs but then anomalous MAC traffic on the network with duplicate IPs would lead me more towards a joint MAC/ARP poisoning than rogue DHCP. Especially since the OP mentioned the servers are using static IPs which would not check with any DHCP server and would only report duplicate IPs if an ARP was detected.

That being said, the details provided by the OP dont paint a very clear picture. Windows/Linux generally only do a duplicate IP check once and that is done when binding the IP. They only bind the IP when the interface state changes either due to reboot of the server or switch, restarting networking or interfaces, disconnecting/reconnecting cables and/or adding/removing IPs.

In my experience, once the IP is bound, the device owns that IP and doesnt usually try to detect duplicates. It usually is the 2nd device to try and claim that IP that throws duplicate IP errors while the 1st device continues happily working on that IP address with no error message; though it may experience some packet loss due to ARP issues at the gateway...

It's not clear from the OPs post what devices are reporting duplicate IPs and, assuming they are the original owners of the IP, what network state change may have occurred to cause it to rebind the IP.

As far as the not being able to ping things on the network, that would make sense since a duplicate IP cant be bound and, assuming there are no other IPs (or they are all flagged duplicate) on the device, a device without a bound IP is not addressable on the network.

It sounds like there may be some more serious issues in the network, especially since moving some cables about seems to have solved the issue...

Perhaps a spanning tree problem? A race situation could give off the appearance of a Packet/MAC/ARP flood and would eventually cause the switch to reboot which would cause the interface to rebind the IP thus getting the duplicate.

Or maybe there was a device actually performing a MAC flood? After moving cables, the device may now be disconnected or otherwise isolated or maybe after restarting the interface by disconnecting/reconnecting the cable, it stopped spamming. The MAC flood would cause a CAM table overflow that again could cause the switch to reboot which would cause the interface to rebind the IP thus getting the duplicate.

But that's all pure speculation because we dont have a clear understanding of exactly what was experienced or how things are interconnected before/after things subsided.

it_consultant

I am not saying a MAC flood COULDN'T cause duplicates, I just haven't seen it. I have been proven wrong before so if you have seen this thing play out then I defer to your experience. One of the problems with OP's scenario is that the affected computers were attached to non-managed switches, so we were really shooting in the dark.