ARP issue

DevilWAH · January 2015

HI can any one tell me what is causing this arp entry to appear (this is on a 2960X Cisco switch

Current configuration : 82 bytes!
interface Vlan666
 ip address 20.20.255.50 255.255.255.0
 no ip proxy-arp
end


DP1-North-A#sh run int vlan 1 
Building configuration...


Current configuration : 48 bytes
!
interface Vlan1
 no ip address
 shutdown
end


DP1-North-A#sh arp
DP1-North-A#sh arp 10.10.224.198
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.224.198         2   0050.56ae.0aab  ARPA   Vlan1
DP1-North-A#
DP1-North-A#

While it is showing up as seen on vlan 1, then I can't connect to the device .198. but as the device only has an interface in vlan 666 and its default gate way is set to use this interface. Why is i learning any ip's on vlan 1?

Cheers

Edit,

10.10.x.x is an ip address in vlan 1 where it is expected to be.

d4nz1g · January 2015

You just received an arp reply on an interface associated to vlan1.

gorebrush · January 2015

I thought that MAC looked familiar - VMware...

DevilWAH · January 2015

gorebrush wrote: »

I thought that MAC looked familiar - VMware...

Indeed

and i think that is the issues, VMware sends a reverse proxy request when vmotion or other things happen to alert the network that its mac address has changed.

[QUOTE=d4nz1g ]You just received an arp reply on an interface associated to vlan1. [/QUOTE]

Only Layer 3 interfaces should respond to an arp, as the switch only has a layer 3 interface in vlan 666 active, (the one in vlan 1 being shut down), it should not be responding to ARP's in vlan 1. its only way to talk to the rest of the network is via the DFGW in vlan 666.

So while it does not have the ARP entry in vlan 1, it can talk to 10.10.244.198 by going out of interface 666 to the dfgw 20.20.255.1 which can route the traffic to the vlan 1 subnet.

as soon as the arp entry is there it can no longer talk to the device as it now tries to check if it can get to the IP address directly, sees the arp entry, but without and active interface in vlan1 it all fails.

So this is not simple the switch reacting to a arp request (that it could never have sent, as above its VMWare sending an unsolicited ARP), its acting on it when it should not.

EdTheLad · January 2015

The switch is not sending an arp request, it has only received an arp reply and cached the entry. This is called promiscuous arp, cisco layer 3 interfaces send an arp reply as soon as they are brought up, you just happen to be connected on a device on vlan 1 thats sending promiscuous arp. Cisco's implementation of promiscuous arp uses arp replies rather than arp requests!

d4nz1g · January 2015

This could happen even if the SVI is shutdown? Never seen it before.

DevilWAH · January 2015

But as mentioned the layer 3 interface on the switch in vlan 1 never comes up. its in an admin down state.

I get how the arps work, but why does a admin down interface respond to them? This is not correct, a shut down interface should not send or recive arp requests.

EdTheLad · January 2015

DevilWAH wrote: »

I get how the arps work, but why does a admin down interface respond to them? This is not correct, a shut down interface should not send or recive arp requests.

As i said previously, the interface is not responding to arp, it is receiving an arp reply, just because its getting an arp reply does not mean its sending an arp request. As to why the switching is recording the arp reply for an SVI that's down, that may or may not be default behavior, i've never noticed as i normally wouldn't be checking the arp cache for an entry on a vlan without an up SVI.
Maybe, once there is one active SVI on a switch, it will record all arp messages even if no SVI is up for a particular vlan, it's quite an easy test to setup.

DevilWAH · January 2015

EdTheLad wrote: »

As i said previously, the interface is not responding to arp, it is receiving an arp reply, just because its getting an arp reply does not mean its sending an arp request. As to why the switching is recording the arp reply for an SVI that's down, that may or may not be default behavior, i've never noticed as i normally wouldn't be checking the arp cache for an entry on a vlan without an up SVI.
Maybe, once there is one active SVI on a switch, it will record all arp messages even if no SVI is up for a particular vlan, it's quite an easy test to setup.

Well it cant work, of the switch has an arp entry in a vlan that it does not have an interface in, it cant then talk to that device. The reason this came about is that the switches keep losing connection to the the management and authentication servers. They are on vlan 1 while the switches management interface is in vlan 666. So when I was trouble shooting the issue I found the ARP entry present.

However the fact still remains an arp entry on device when there is no active L3 interface in the vlan will cause a loss of communication. As I mentioned we know that vmware sends gratuitous ARP's to update the network that it has changed its IP address. This should only be of interested to later 3 devices on the same vlan segment. IF the switch does not have a interface active in the vlan then it should not have any ARP entries from that vlan, as there are no cases where it would need to know them, it should always be using its DFGW on its actvie interface to reach devices in disabled vlans.

no one is questioning what is happening, the question is why it would it be allowed to happen, and more importantly how to prevent it.

slinuxuzer · January 2015

Vlan 666 is the problem

, Ok now that thats out of my system.

Vlan1 is usually the default untagged/native vlan interface, are you using vlan 1 for data traffic? Is vlan1 your default native/untagged vlan on these uplinks? I am thinking that maybe there is a vSwitch that is sending CDP advertisements from this mac. Check your vSwitches for CDP settings and see if your sending CDP, if you are try disabling this and see if you stop seeing this ARP entry. CDP uses the native vlan.

Also do show cdp neighbors from the switch and see if you see vmnic entries in the CDP table, that will tell you if your getting CDP advertisements from VMware.

DevilWAH · January 2015

slinuxuzer wrote: »

Vlan 666 is the problem , Ok now that thats out of my system.

Vlan1 is usually the default untagged/native vlan interface, are you using vlan 1 for data traffic? Is vlan1 your default native/untagged vlan on these uplinks? I am thinking that maybe there is a vSwitch that is sending CDP advertisements from this mac. Check your vSwitches for CDP settings and see if your sending CDP, if you are try disabling this and see if you stop seeing this ARP entry. CDP uses the native vlan.

Also do show cdp neighbors from the switch and see if you see vmnic entries in the CDP table, that will tell you if your getting CDP advertisements from VMware.

Oh i had not though of CDP being the culprit, but yes VMware does talk CDP to the network... thats an idea i shall look in to.

I have always used 666 for management and 999 for black hole vlan gets a few raised eyebrows from customers when i present design documents. some have actually requested to use a different number

WE are migrating from vlan 1 but there is a lot of legecy system meaning it is not a quick swap.

slinuxuzer · January 2015

Keep us posted, I'd be interested to know whats up with this. Thanks.

EdTheLad · January 2015

DevilWAH wrote: »

no one is questioning what is happening, the question is why it would it be allowed to happen, and more importantly how to prevent it.

DevilWAH wrote: »

Only Layer 3 interfaces should respond to an arp, as the switch only has a layer 3 interface in vlan 666 active, (the one in vlan 1 being shut down), it should not be responding to ARP's in vlan 1.

As per your initial post, your understanding was the switch was replying to arp, which it is not! Good luck, you don't seem to need help.

DevilWAH · January 2015

I said "Respond" not "Replying", by respond I mean the switch was talking an action apon receiving an arp packet, solicited or unsolicited.

The only time it has been suggested that the switch was sending any type of arp packet was "This is called promiscuous arp, cisco layer 3 interfaces send an arp reply as soon as they are brought up"

DevilWAH · February 2015

slinuxuzer wrote: »

Keep us posted, I'd be interested to know whats up with this. Thanks.

gets more interesting.

Two switch stacks with identical config (different management address of course), and the ip that is getting added to the arp table is that of the CISCO Prime appliance, and it happens when ever the prome server connects to the switch, it might be a config archive or deploying a config. But it connects, the arp entry gets entered and it can no longer communicate with the Prime server.

I dont see any code being run that is putting the ARP entry there, but it seems to be restricted to this one IP address at the moment. The other monitoring server that is also a VMWare appliance does not have the same issue.

d4nz1g · February 2015

Do you have any capture of this traffic?

DevilWAH · February 2015

I have looked at capturing it, but to be honest there seems to be no difference in traffic between working and non working devices. Because this is so intermittent happening what seems randomly (not every time the prime server connects) there seems to be no difference in traffic flow.

networker050184 · February 2015

Possibly an ARP in the wrong VLAN bug. Could also be the device tagging and something like EdTheLad describing.

DevilWAH · February 2015

Anoyingly this only happens on a single switch stack now!

6506 with two ports uplinked to each of 5 X 2960x stacks

Stack 1 - 4 run with out issues, its just the 5th stack that keeps doing it. The only difference is (and I have compared the config with text compare tools) the Management IP address and that the stack not working has 4 switches in it while the rest have 2.

The rest seem to be fine now, I am thinking i might split it in to 2 stacks of 2 and see what happens. The onyl other things before that is that in the stack of 4 the 3rd switch has become the master. Normally I would not bother about this, but to keep in line with the rest I have forced switch 1 to be primary, just in case some strange bug in the code does not like it when SW1 is not master.

if all else fails I will down grade the config to what it was a few weeks back before this issues started to happen.

DevilWAH · February 2015

Well that did not go as expected!

Set switch one to a higher priority and reloaded the stack (to get a fresh start). Now individual members are bouncing up and down. one moment there is 4 switches in the sack, the next only 2, then they come back and another one will drop off.

Not good!!

d4nz1g · February 2015

Are they stacked correctly?
Check the stack cables ( I have had some issues with stack cabling long ago).

Never touched a 2960x before, have you checked the bug tracker for this model?

Also, look around for any logging/debug information. Could be very helpful.

DevilWAH · February 2015

Feb 3 11:22:50.546: %PM-3-INTERNALERROR: Port Manager Internal Software Error (pm_vlan_test_portlist(vlan, pd->globalNumber): ../switch/pm/pm_vlan.c: 1781: pm_vlan_rem_port) (DP1-West-2)
-Traceback= 5ABD50z 30D0A1Cz 3147A24z 3136CCCz 3137868z 24823F0z 3126090z 30D4980z 2E907C8z 2E90A8Cz 2FD1A68z 29BC194z 2FCA69Cz 2E8CA08z 2E8DD8Cz 3433D00z (DP1-West-2)
Feb 3 11:22:50.546: %PM-3-INTERNALERROR: Port Manager Internal Software Error (pm_vtpvlan_bitlist_test(&pd->operInfo.trunkVlans, vlan): ../switch/pm/pm_vlan.c: 1793: pm_vlan_rem_port) (DP1-West-2)
-Traceback= 5ABD50z 30D0A1Cz 3147B04z 3136CCCz 3137868z 24823F0z 3126090z 30D4980z 2E907C8z 2E90A8Cz 2FD1A68z 29BC194z 2FCA69Cz

Seems to suggest it is to do with DOT1x configuration. and there are some bugs but not specific to this model.

The ARP learning on vlan 1 is also not specific to vmware clients. Just also happened on the PC i was connected from. Nothing in logs, for about 30 min I was happily working away. Then just got disconnected, jumped back on from another box and see my IP address in the arp cache assigned to VLAN1. Cleared it and I can connect again.

DevilWAH · February 2015

Right well.

I was looking on the switch FS and I could see crash **** and a number of files that where not on other switch stacks. I also was seeing PAGP flapping errors (even though LACP was the ethen channel in use). Now I know 2 of the switchs in this stack where taken from other existing stacks and I found out the person who moved them had only done a "write erase" but had not cleared the VTP / VLAN data.

Not being sure exactly what they had done I thought I would take the whole stack back to Factory defaults and rebuild. So copied running config to a TFTP server, Formated the flash on all 4 devices, erased the nvram. Copied the same IOS I had been using back to all 4 flashes and booted the first switch.

Coped the config exactly as was back to the switch and made sure it seemed stable and configure VTP again, then one by one powered up the other three and let them join the stack.

Result, no issues, no strange ARP entries, crash files, network dropping, no etherchannel errors. In fact its been stable for 48 hours now. So all I can image is that when the two switches where original added, the existing config conflicted and the stack was not full syncing.

Thanks for all the comments but fingers crossed I think this is all OK now.

Oh the only difference is that I am not using the tar image with the web elements now, just the bin file. but the other 4 stacks have the full tar image on and are running with out issues.

ARP issue

Comments