How to troubleshoot intermittent interVLAN routing issues

--chris--

I have run into an issue that pops up 1-2x a week for 40 to 180 minutes. It appears as thought all routing between VLAN's just drops out. I can ping hosts within the same VLAN, but I am unable to ping hosts in other VLANS. This of course means internet access is down as well (I have tested that; the ISP connection past the demarc point is good...its also able to get through the firewall, the connection "drops" out at the next hop past the firewall which is a Dell 3548 L3 switch)

If I change my laptops IP config to match a different VLAN, then I can ping the hosts in that VLAN but again I can not ping past the gateway.

So far this issue has pop'd up twice and it is starting to give me gray hair. I know just enough about switching/routing to start working on the issue, but my core knowledge really does not extend into troubleshooting.

Any suggestions on where to begin with this issue? Any debugs that could be suggested?

VAHokie56

So I am guessing the Dell switch is doing all the routing for LAN sub-nets and may has a default route pointing to you internet router or fW?

--chris--

VAHokie56 wrote: »

So I am guessing the Dell switch is doing all the routing for LAN sub-nets and may has a default route pointing to you internet router or fW?

Yes; thank you for verbalizing what I could not.

The Dell L3 has a default route of 0.0.0.0/0 via 10.1.0.254 which is the firewall/internet gateway. Knowing this then, the L3 switch has to have at least one trunk port with sub-interfaces on it that are acting as the gateway for each VLAN right?

VAHokie56

I think more likely ( and mind you I have never worked on a dell switch ) that it has SVI's for each vlan making them routes local to the switch or "connected" and they are in the routing table...the link to the router I would guess is a routed switch port off the the dell with a IP address...if you want to PM with config I can try and help you just make sure you ***** out and Passwords or public IP's.
I really think it sounds like the switch is taking a fart every so often though and it most likely may not be a config issue...I assume the dell can log sys messages , do you have any from the time in question?

it_consultant

^^ This ^^

Chris you are thinking of a router on a stick which is old school. Most likely the Dell's have a virtual router which will make this unnecessary. In my stack, which isn't Dell, I have a couple of virtual routers (in Brocade it is called VE or 'virtual ethernet) which I configure in each VLAN. When I configure those they automatically route for the VLAN to other VEs in the stack assuming that there is at least one untagged (or 'access') port active on the VLAN. Plugged into the firewall is a port which is untagged in one of those VLANs and then a simple 0.0.0.0 0.0.0.0 route to the firewall port. I think the Dell's are configured similarly.

Seeing the running config will help greatly.

--chris--

This makes sense. SVI's right?

This is the running config of the dell 3548 L3 which is the next hop after the firewall:

SW_SOUTH# show run
spanning-tree mode rstp
interface range ethernet e(30-47)
spanning-tree portfast
exit
interface ethernet e12
description DELL_PC_NEW_AUGUST_2014
exit
interface ethernet e26
description SNIFFER_COMPUTER
exit
interface ethernet e27
description GRANDSTREAM
exit
interface ethernet e29
description FIREPANEL
exit
interface range ethernet e(30-31,33)
description PHONES
exit
interface ethernet e32
description PHONE_R_X112
exit
interface range ethernet e(34-47)
description FIELDPOINT
exit
interface ethernet e48
description RUCKUS_AP__828
exit
interface ethernet g2
description LINK_TO_NO
exit
interface ethernet g3
description LINK_TO_W
exit
interface range ethernet e(30-33,4,g(2,4)
switchport mode trunk
exit
vlan database
vlan 8,21,52,100-101,104,112,120,128,900,999
exit
interface range ethernet e(34-47)
switchport access vlan 8
exit
interface range ethernet e(30-33)
switchport trunk native vlan 8
exit
interface range ethernet e48,g(2,4)
switchport trunk allowed vlan add 8
exit
interface range ethernet e(30-33,4,g(2,4)
switchport trunk allowed vlan add 21
exit
interface ethernet e27
switchport access vlan 52
exit
interface range ethernet e48,g(2,4)
switchport trunk allowed vlan add 52
exit
interface range ethernet e(30-33,4,g(2,4)
switchport trunk allowed vlan add 100
exit
interface range ethernet e(30-33,4,g(2,4)
switchport trunk allowed vlan add 101
exit
interface ethernet e48
switchport trunk native vlan 104
exit
interface range ethernet e(30-33),g(2,4)
switchport trunk allowed vlan add 104
exit
interface range ethernet e(30-33,4,g(2,4)
switchport trunk allowed vlan add 112
exit
interface range ethernet e(30-33,4,g(2,4)
switchport trunk allowed vlan add 120
exit
interface range ethernet e(30-33,4,g(2,4)
switchport trunk allowed vlan add 128
exit
interface ethernet g4
switchport trunk allowed vlan add 900
exit
interface ethernet g3
switchport access vlan 999
exit
interface range ethernet e(30-33,4,g(2,4)
switchport trunk allowed vlan add 999
exit
interface vlan 101
name SWITCH_MGMT
exit
interface vlan 900
name FIELDPOINT
exit
voice vlan oui-table add 0001e3 Siemens_AG_phone________
voice vlan oui-table add 00036b Cisco_phone_____________
voice vlan oui-table add 00096e Avaya___________________
voice vlan oui-table add 000fe2 H3C_Aolynk______________
voice vlan oui-table add 0060b9 Philips_and_NEC_AG_phone
voice vlan oui-table add 0080f0 PANASONIC______________
voice vlan oui-table add 00d01e Pingtel_phone___________
voice vlan oui-table add 00e075 Polycom/Veritel_phone___
voice vlan oui-table add 00e0bb 3Com_phone______________
voice vlan id 52
voice vlan cos 5
interface ethernet e30
voice vlan enable
exit
interface ethernet e31
voice vlan enable
exit
interface ethernet e32
voice vlan enable
exit
interface ethernet e33
voice vlan enable
exit
interface vlan 999
ip address 10.1.0.240 255.255.255.128
exit
interface vlan 101
ip address 10.1.101.32 255.255.255.0
exit
interface vlan 1
ip address 192.168.1.240 255.255.255.0
exit
interface vlan 900
ip address 192.168.50.253 255.255.255.0
exit
ip default-gateway 10.1.101.1
hostname SW_SOUTH
logging 10.1.0.10
username -- password -- level 15 encrypted
snmp-server community -- rw view --
clock timezone -5 zone EDT
clock source sntp
sntp unicast client enable
sntp unicast client poll
sntp anycast client enable
sntp broadcast client enable
sntp server -- poll
sntp server 2610:20:6f15:15::27 poll
ip name-server 8.8.8.8
snmp-server set rlEventsDeleteEvents rlEventsDeleteEvents 1


Default settings:
Service tag:


SW version


Fast Ethernet Ports
==========================
no shutdown
speed 100
duplex full
negotiation
flow-control off
mdix auto
no back-pressure


Gigabit Ethernet Ports
=============================
no shutdown
speed 1000
duplex full
negotiation
flow-control off
mdix auto
no back-pressure


interface vlan 1
interface port-channel 1 - 15


spanning-tree
spanning-tree mode STP


qos basic
qos trust cos

___________________



Interesting note:

I was talking to an employee that sits near the network closet where the firewall for this location is, the L3 switch (next hop after firewall) and another l2 switch. He said the APC battery backup that all of this stuff is plugged into went off (Audio alarm) yesterday. He went in and pushed the button to make the noise stop. He said this turned off everything (firewall, L3 & L2 switch) so he pushed the button and everything came back up. He said this was at about 2:10 PM. This is the exact time the network came "back up" yesterday. Coincidence?

--chris--

I checked for the logs, but because of the end user power-cycling it yesterday there is nothing from before the power-cycle event. When I view the logs all I can see is it powering up, ports checking in then my remote connection getting created...

it_consultant

I have a couple of suggestions based on what I see. When I initially looked at this I thought this was a dell stack created by using the stacking ports available on the Dell, but what I am seeing (because of the trunk ports 30-33 etc) is that you are probably plugging in an aggregation or access layer into these switches. Now I am thinking STP and or loops that are introduced at the access layer.

This may not solve the problem but I would convert to 802.1W RSTP across the switching fabric. I would declare all of the access ports as spanning tree access mode (it might be a different term in dell) but that will essentially prevent a switch added on an access port from participating in STP. I would then declare your trunk ports as spanning tree point to point links which are ports allowed to participate in spanning tree topology updates. I would then put a BPDU filter on all the access ports which will prevent a looped switch at someone's desk from spewing a ton of BPDUs into your aggregation and core.

Doing that will at least prevent one of the most common scenarios which cause "network down" situations. Beyond that, I would take a look at the release notes for the newer Dell firmwares (yours are from 2009) and see if a problem similar to yours is called out as a defect. This actually happened to me, I have hardened switches which would go to 100% utilization after a PCI scan. The next firmware released fixed that defect. My core switches went to 100% utilization if you hit a certain keystroke in the console - the VRRP service took all the CPU even though we weren't using it. Sometimes there are good things to be found in those releases.

VAHokie56

Is it just one L2 switch? kind of sounds like the APC maybe died nucked your L3 switch ( bye bye inter-vlan routing) the L2 switch at this point would still allow communication to stuff on same sub-net

--chris--

VAHokie56 wrote: »

Is it just one L2 switch? kind of sounds like the APC maybe died nucked your L3 switch ( bye bye inter-vlan routing) the L2 switch at this point would still allow communication to stuff on same sub-net

I left out some network details to limit the scope of my original question.

Your suggestion would be logical but the l3 switch is one of 3 and there are 2 L2 switches in there as we'll ( all of this is spread across 3 buildings). The APC, one L3 and firewall are in building 2. I was in building 1 with another L3 switch and was also unable to ping out of any vlan. Same story in building 3.

VAHokie56

doesn't really make sense for there to be three L3 switches IMO...two I can live with for FHRP's but I dont see any HSPR or GLBP on that dell switch (my dell skills suck thought). Is it possible you have three L3 switches but only one (the one you posted config from) is actually handling routing? and if so is the one above connected tot he jacked up APC?

phoeneous

1. Is this a new design?
2. Did the problem just start happening or has it been happening since day one?
3. Have you checked the logs in the UPS?

Always start at the physical later and in this case would be the ups. Plug the ups comm cable to a pc and install the ups software so you can check it. L2 or l3 troubleshooting wont do much if power is wonky.

--chris--

VAHokie56 wrote: »

doesn't really make sense for there to be three L3 switches IMO...two I can live with for FHRP's but I dont see any HSPR or GLBP on that dell switch (my dell skills suck thought). Is it possible you have three L3 switches but only one (the one you posted config from) is actually handling routing? and if so is the one above connected tot he jacked up APC?

See the quick and dirty physical diagram I made, does this configuration make sense now? The L3 we have been discussing is not the only L3 that performs routing.

Yes, the config posted above is the one actually connected to the APC that had an alarm go off.

--chris--

phoeneous wrote: »

1. Is this a new design?
2. Did the problem just start happening or has it been happening since day one?
3. Have you checked the logs in the UPS?

Always start at the physical later and in this case would be the ups. Plug the ups comm cable to a pc and install the ups software so you can check it. L2 or l3 troubleshooting wont do much if power is wonky.

1) No, this design has been in place since August of this year.
2) This issue started on the 6th of this month.
3) I have not thought to check the logs in the UPS, I will be Monday though. Thats a great idea, thanks!

VAHokie56

lol this is so confusing...OK so if there is L3 in every building then there has to be routing between them also, that means L3 boundary between each building meaning that vlans are pretty much local to each building as well so there would be no way users in diff building could be on the same sub-net. Would you say that's an accurate statement? now if you tell me you have L3 trunks between the buildings and are doing routing and L2 across them all , I would say rip it all out and start over because that's ugly...also which building has the bad APC?

--chris--

VAHokie56 wrote: »

lol this is so confusing...OK so if there is L3 in every building then there has to be routing between them also, that means L3 boundary between each building meaning that vlans are pretty much local to each building as well so there would be no way users in diff building could be on the same sub-net. Would you say that's an accurate statement? now if you tell me you have L3 trunks between the buildings and are doing routing and L2 across them all , I would say rip it all out and start over because that's ugly...also which building has the bad APC?

Yes, building 1,2,3 each have a L3 switch. Building 2 has a 3560, so if your more comfortable with Cisco I could do a sh run on that and post it here. The L3's have static routes that point to each other.

There are two VLANs that are in all three buildings (VoIP & "admin" aka office PC's) i.e. some users in building 1-2-3 are in the same VLAN.

Building 1 has the "bad" APC which I will be looking into tomorrow.

I agree on rebuilding the network, its very complicated and it does not need to be. We have customers 3-4X this size on 1 to 3 VLANs, not a dozen. However, the customer has short term goals that need to be met and that is not one of them.

it_consultant

It would surprise me if the routing was the problem, we would expect that to be a constant problem. I am still at spanning tree and people looping switches but it is also possible that you have a VLAN leak. Since you are running a configuration where some VLANs are end to end and some are not I wonder if the VLAN config is messed up on the links somewhere. When you say "switchport mode trunk" and then declare the allowed VLANs, that doesn't necessarily remove all other VLANs from using the trunk. It might in Dell but my experience is that once you say "mode trunk" then all VLANs are allowed to cross the port (hence the term 'trunk') and you prune. From what I see the pruning is half done.

phoeneous

it_consultant wrote: »

When you say "switchport mode trunk" and then declare the allowed VLANs, that doesn't necessarily remove all other VLANs from using the trunk.

Huh? That's exactly what that command does.

Catalyst 2960 and 2960-S Software Configuration Guide, 12.2(55)SE - Configuring VLANs [Cisco Catalyst 2960 Series Switches] - Cisco

it_consultant

He is not using a Cisco - well, one of them is a Cisco, the rest are Dells. BTW, the reference you sent actually doesn't say that. Reference:

"By default, a trunk port sends traffic to and receives traffic from all VLANs. All VLAN IDs, 1 to 4094, are allowed on each trunk. However, you can remove VLANs from the allowed list, preventing traffic from those VLANs from passing over the trunk. To restrict the traffic a trunk carries, use the switchport trunk allowed vlan remove vlan-list interface configuration command to remove specific VLANs from the allowed list."

In his pasted config he does not have the 'allowed vlan remove xxx' he just has 'allowed vlan xxx', hence my conclusion that the trunks are not properly pruned. What I am unsure of is that in the absence of the "remove vlan list' command, does explicitly allowing certain VLANs implicitly block the other VLANs that are allowed by default. I do not know that to be true.

--chris--

it_consultant wrote: »

He is not using a Cisco - well, one of them is a Cisco, the rest are Dells. BTW, the reference you sent actually doesn't say that. Reference:

"By default, a trunk port sends traffic to and receives traffic from all VLANs. All VLAN IDs, 1 to 4094, are allowed on each trunk. However, you can remove VLANs from the allowed list, preventing traffic from those VLANs from passing over the trunk. To restrict the traffic a trunk carries, use the switchport trunk allowed vlan remove vlan-list interface configuration command to remove specific VLANs from the allowed list."

In his pasted config he does not have the 'allowed vlan remove xxx' he just has 'allowed vlan xxx', hence my conclusion that the trunks are not properly pruned. What I am unsure of is that in the absence of the "remove vlan list' command, does explicitly allowing certain VLANs implicitly block the other VLANs that are allowed by default. I do not know that to be true.

I to am still focusing on the noisy APC and possible VLAN issues. I am more focused on the L1 than than the L2/L3 because this network has been fine for 3 months, no changes at all then two weeks ago it had that outage (no outage since either, knock on wood).

I had food poisoning or something yesterday, but today I will be able to check out the APC's logs. Hopefully that has a big red flag in it.

phoeneous

it_consultant wrote: »

He is not using a Cisco - well, one of them is a Cisco, the rest are Dells. BTW, the reference you sent actually doesn't say that. Reference:

"By default, a trunk port sends traffic to and receives traffic from all VLANs. All VLAN IDs, 1 to 4094, are allowed on each trunk. However, you can remove VLANs from the allowed list, preventing traffic from those VLANs from passing over the trunk. To restrict the traffic a trunk carries, use the switchport trunk allowed vlan remove vlan-list interface configuration command to remove specific VLANs from the allowed list."

In his pasted config he does not have the 'allowed vlan remove xxx' he just has 'allowed vlan xxx', hence my conclusion that the trunks are not properly pruned. What I am unsure of is that in the absence of the "remove vlan list' command, does explicitly allowing certain VLANs implicitly block the other VLANs that are allowed by default. I do not know that to be true.

The reason why add and remove are not in the config is because theyre dynamic commands. Sure 1 to 4094 are allowed on the trunks by default until you specify otherwise. Why else would the allowed vlan command even exist if you had no control over which vlans traversed the trunk??

Lab it up, capture some traffic, you'll see.

it_consultant

On my switches you see both the adding of allowed VLANs and the removing of the non-allowed VLANs; which we should all be doing for at least VLAN 1, which OP doesn't have. However, the idea that we are implicitly denying other VLANs because we have allowed some VLANs is not evident to me unless I see it documented. To me this is like having an allow rule in a firewall without a catch all deny any any at the end of the ruleset. I am not saying you are wrong I am saying that the documentation does not tell me that allowing specific VLANs then denies all others; unless I just didn't see it.

networker050184

Phoeneous is correct. When you specifically allow certain VLANs the rest are removed on a Cisco switch. Just do a show interface trunk command to see for yourself. It will list all allowed VLANs. Not sure about Dell though.

it_consultant

Believe it or not I do not have a Cisco switch sitting around! I have HP and Brocade so I believe you. I do it both ways on my switches, maybe I don't have too. There is more than one way to skin a cat and I suppose more than one way to prune a trunk.

--chris--

The APC's logging is hosed, its time & dates appear to reset every 24 hour. Every log entry is dated Dec 31 1999.

I am putting in a new battery backup tomorrow, and sorting out the other garbage as we move along.

Thanks to all the other contributors, I learned that pruning VLAN trunks can be done a few ways