Unable to ping hosts from switches and hosts

Genji

I'm currently building a network which kind of pushes the limits of CCNA and have gotten stuck on something that feels so easy that I now feel rather silly...

The network:

I have a Fibreglass Stack Core comprising of 3x 3750-X-24S-S switches.
I have a Copper based Stack Core comprising of 2x 3750-X-48T-S switches.
I have 8 standalone 2960-S-48LPD-L switches.
I have 6 standalone 2960-S-24LPD-L switches.

All switches have a Fibreglass trunk running into the fibre stack core.
the copper based stack core has a TenGig trunk, the rest 1Gig

There are 4 VLANs:

vlan 1 -> default -> Production
vlan 25 -> OOB -> Out Of Band Management
vlan 50 -> iSCSI
vlan 192 -> Scanner -> Wireless Scanning network

Zooming in on the part of the network I have an issue with:

I have one of the standalone 2960 switches (SW003) trunked over fibre into the 3750-X fibre switch (SW002). In turn, there is another fibre trunk coming out of this 3750 going into another standalone 2960 (SW052).

So, the issue:

I have 2 laptops.
LT1 is connected to SW003, LT2 to SW052
The switches cannot ping the laptops
The laptops cannot ping eachother

My Troubleshooting:

I have completely disabled the windows firewall service (As our corp FW policy dictates an ICMP block on unknown networks) on both laptops.
I have assigned interface vlan ips to vlan 1 on each switch the traffic should go through (This is not standard practice with us as it should work without int vlan ip address configured however for the sake of TS and checking where traffic stops, I added it in there anyway)
After doing this, the laptops can now ping all switches that have the int vlan 1 ip address configured. Further; The switches can ping eachother on their OOB and VLAN1 IPs. The issue however still remains.
I have checked and double checked my trunk ports and encapsulation types.
I have ensured the access ports the laptops are hooked into are set to access vlan 1 (although this should be default ootb functionality)
Kind of at a loss at what I've missed now..

SW003 running-config:

Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(55)SE7, RELEASE SOFTWARE (fc1)

interface range GigabitEthernet1/0/1-44
 description User Access
 switchport mode access
 spanning-tree portfast

interface GigabitEthernet1/0/49
 description Fiber Uplink
 switchport trunk allowed vlan 1-220
 switchport mode trunk
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 mls qos trust cos
 auto qos trust

interface Vlan1
 description Production
 ip address 172.25.44.3 255.255.254.0

SW003 sh ip int br and sh vlan:

Vlan1                       172.25.44.3     YES manual up                    up
GigabitEthernet1/0/5   unassigned      YES unset  up                    up
GigabitEthernet1/0/49  unassigned      YES unset  up                    up

1    default                          active

SW002 running-config:

Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 12.2(5[IMG]https://us.v-cdn.net/6030959/uploads/images/smilies/icon_cool.gif[/IMG]SE2, RELEASE SOFTWARE (fc1)

interface range GigabitEthernet1/0/1-24
 description Fiber Uplink
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1-220
 switchport mode trunk
 srr-queue bandwidth share 1 30 35 5
 queue-set 2
 priority-queue out
 mls qos trust cos
 auto qos trust

SW002 sh ip int br and sh vlan:

Vlan1                       172.25.44.2     YES manual up                    up
GigabitEthernet1/0/1   unassigned      YES unset  up                    up

1    default                          active

SW052 running config:

Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(55)SE7, RELEASE SOFTWARE (fc1)

interface range GigabitEthernet1/0/1-44
 description User Access
 switchport mode access
 spanning-tree portfast

interface GigabitEthernet1/0/49
 description Fiber Uplink
 switchport trunk allowed vlan 1-220
 switchport mode trunk
 srr-queue bandwidth share 1 30 35 5
 priority-queue out
 mls qos trust cos
 auto qos trust

interface Vlan1
 description EDC Production
 ip address 172.25.44.52 255.255.254.0

SW052 sh ip int br and sh vlan:

Vlan1                        172.25.44.52   YES manual up                    up
GigabitEthernet1/0/19  unassigned      YES unset  up                    up
GigabitEthernet1/0/49  unassigned      YES unset  up                    up

Laptops are set to static ips of:
LT2 172.25.44.21/23 -> DG: 172.25.44.52
LT1 172.25.44.31/23 -> DG: 172.25.44.03

ping results:

from SW003:

SW003>ping 172.25.44.02


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/21 ms
SW003>ping 172.25.44.52


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.52, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/5 ms
SW003>ping 172.25.44.21


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.21, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
SW003>ping 172.25.44.31


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.31, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

from SW002:

SW002>ping 172.25.44.03


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/9 ms
SW002>ping 172.25.44.52


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.52, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW002>ping 172.25.44.21


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.21, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
SW002>ping 172.25.44.31


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.31, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

from SW052:

SW052#ping 172.25.44.03


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/5 ms
SW052#ping 172.25.44.02


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/16 ms
SW052#ping 172.25.44.21


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.21, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
SW052#ping 172.25.44.31


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.31, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

from Laptop 1:

LT1>ping 172.25.44.03


Pinging 172.25.44.3 with 32 bytes of data:
Reply from 172.25.44.3: bytes=32 time=5ms TTL=255
Reply from 172.25.44.3: bytes=32 time=2ms TTL=255
Reply from 172.25.44.3: bytes=32 time=1ms TTL=255
Reply from 172.25.44.3: bytes=32 time=1ms TTL=255


Ping statistics for 172.25.44.3:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 5ms, Average = 2ms


LT1>ping 172.25.44.02


Pinging 172.25.44.2 with 32 bytes of data:
Reply from 172.25.44.2: bytes=32 time=8ms TTL=255
Reply from 172.25.44.2: bytes=32 time<1ms TTL=255
Reply from 172.25.44.2: bytes=32 time=5ms TTL=255
Reply from 172.25.44.2: bytes=32 time=1ms TTL=255


Ping statistics for 172.25.44.2:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 8ms, Average = 3ms


LT1>ping 172.25.44.52


Pinging 172.25.44.52 with 32 bytes of data:
Reply from 172.25.44.52: bytes=32 time=1ms TTL=255
Reply from 172.25.44.52: bytes=32 time<1ms TTL=255
Reply from 172.25.44.52: bytes=32 time<1ms TTL=255
Reply from 172.25.44.52: bytes=32 time<1ms TTL=255


Ping statistics for 172.25.44.52:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 1ms, Average = 0ms


LT1>ping 172.25.44.21


Pinging 172.25.44.21 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.


Ping statistics for 172.25.44.21:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

---

I realize this first post is a far cry from an ideal introduction and a 'hey how is everyone?' however I would much appreciate any input or help from anyone.

Sorry it's so full of info and details, I'm hoping I covered everything to head off some questions.

Thanks in advance!

atorven

I don't see a native vlan defined on those trunks. Also, why do you have 2 different default gateways for the same subnet?

xnx

You need a matching native VLAN on each side

Genji

Thanks for the quick replies atorven and xnx!

OK,

atorven wrote: »

why do you have 2 different default gateways for the same subnet?

Once I posted the above, I wondered the same too... must've been a brain fart. DG's on both LT's have been changed to 172.25.44.02 (Which is the glass switch stack that both SW003 and SW052 are connected to)

I currently only have the switching environment up, there are no routers, FWs, ASAs, DC's or WAN optimizers connected. Additionally, I don't have DHCP setup on any switches so I have manually assigned the IPs on each LT.

So, running with native vlans - I figured native vlans only really have to be configured if the native vlan is to be anything other than vlan 1.
To make sure I followed your advice, went ahead and did a switchport trunk native vlan 1 on all 3 trunk ports (SW002, SW003 and SW052).
No difference.

Really making sure, I hit the ports the LT's are connected to with the following config (same on both ends):

description User Access
switchport mode trunk 
switchport access vlan 1
switchport trunk native vlan 1
switchport trunk allowed vlan 1
spanning-tree portfast trunk

Still no difference.

Figuring I now must be going mad, I moved LT2 from SW052 into Gi1/0/6 on SW003, still no difference.
I'm struggling to think of anything else I could do to prove it is the switch or the LT's themselves - still up in the air on that one.

Again, thanks for the quick replies, it's well appreciated.

xnx

You need a router to move anything between different VLANS, untagged frames (native VLAN) frames will not need a router

santaowns

ok lets simplify it even more, if the laptops are on same switch can they ping? can switch ping them? this will eliminate possible software firewall and let us focus on the port setups. What are results of "show mac add" - looking for laptop mac. if it shows up on sw003 does sw002 see the mac in its table?

Genji

Hi xnx,

Seeing as I'm only attempting to confirm hosts on vlan 1 can ping each other, I have no need for a router just yet.
The routers I have planned will come into play at a later stage once vlan 1 works to my satisfaction.

santaowns - see below

LT1 and LT2 are now on the same switch:

GigabitEthernet1/0/5   unassigned      YES unset  up                    up
GigabitEthernet1/0/6   unassigned      YES unset  up                    up

The switch still cannot ping either laptop:

SW003>ping 172.25.44.21


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.21, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
SW003>ping 172.25.44.31


Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.25.44.31, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

The laptops can ping the switch but not eachother (example from LT1 - output from LT2 is the same):

LT1>ping 172.25.44.03


Pinging 172.25.44.3 with 32 bytes of data:
Reply from 172.25.44.3: bytes=32 time=8ms TTL=255
Reply from 172.25.44.3: bytes=32 time=2ms TTL=255
Reply from 172.25.44.3: bytes=32 time=1ms TTL=255
Reply from 172.25.44.3: bytes=32 time=1ms TTL=255


Ping statistics for 172.25.44.3:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 1ms, Maximum = 8ms, Average = 3ms

LT1>ping 172.25.44.21


Pinging 172.25.44.21 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.


Ping statistics for 172.25.44.21:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

LT1 and LT2 only use Miscrosoft's Windows Firewall, there is no other firewalling software installed on either Laptop.
The Windows Firewall service has been stopped completely:


We can see both laptops in the mac address-table and arp list on all switches:

SW003>sh mac address-table | inc f01f
   1    f01f.af0d.2278    DYNAMIC     Gi1/0/6
   1    f01f.af48.7888    DYNAMIC     Gi1/0/5

SW003>sh arp | inc 172.25.44
Internet  172.25.44.31            0   f01f.af48.7888  ARPA   Vlan1
Internet  172.25.44.21           26   f01f.af0d.2278  ARPA   Vlan1
Internet  172.25.44.3             -   c414.3cee.ae40  ARPA   Vlan1
Internet  172.25.44.2           219   7426.ac89.4240  ARPA   Vlan1
Internet  172.25.44.52          178   c414.3cee.a5c0  ARPA   Vlan1

SW002#sh mac address-table | inc f01f
   1    f01f.af0d.2278    DYNAMIC     Gi1/0/1
   1    f01f.af48.7888    DYNAMIC     Gi1/0/1

SW002#sh arp | inc 172.25.44
Internet  172.25.44.2             -   7426.ac89.4240  ARPA   Vlan1
Internet  172.25.44.3           177   c414.3cee.ae40  ARPA   Vlan1
Internet  172.25.44.21           25   f01f.af0d.2278  ARPA   Vlan1
Internet  172.25.44.31            0   f01f.af48.7888  ARPA   Vlan1
Internet  172.25.44.52          177   c414.3cee.a5c0  ARPA   Vlan1

SW052>sh mac address-table | inc f01f
   1    f01f.af0d.2278    DYNAMIC     Gi1/0/49
   1    f01f.af48.7888    DYNAMIC     Gi1/0/49

SW052>sh arp | inc 172.25.44
Internet  172.25.44.31            2   f01f.af48.7888  ARPA   Vlan1
Internet  172.25.44.21           28   f01f.af0d.2278  ARPA   Vlan1
Internet  172.25.44.3           180   c414.3cee.ae40  ARPA   Vlan1
Internet  172.25.44.2           221   7426.ac89.4240  ARPA   Vlan1
Internet  172.25.44.52            -   c414.3cee.a5c0  ARPA   Vlan1

atorven

@xnx - From what he posted initially everything that he was trying to reach is in VLAN 1 so no routers are required here.
@Genji - The config that you initially had on your access ports is fine so revert back to that (take off the trunk related configuration) - Add switchport trunk native vlan 1 to all your trunk uplinks (the interfaces that you have labeled as Fiber Uplink)

networker050184

Ok, set the user ports back to access ports. No need for trunk there. No need to set a native VLAN on your trunks either. I would suggest not using VLAN1 for user traffic, but that is a whole other conversation and nothing to do with your problem.

Have you disabled any software firewalls on the hosts? 99% of the time people have this problem that is the solution. If they are both plugged into the same switch on the same VLAN there really isn't much else to it.

atorven wrote: »

Add switchport trunk native vlan 1 to all your trunk uplinks

Why?

xnx

Sorry, i'm not gonna lie I didn't read all your post :P

santaowns

Well if you verify in your ACL that ICMP isn't being blocked, then it has to be a software firewall on the pcs blocking incoming icmp.

Genji

Wow! Those were some fast responses!

atorven/networker050184:
My access ports are back the way they were.
I'm still not getting why I need to mark vlan 1 as native on the trunk but á la, I made it so.

See my response to santaowns above as to software firewalls etc etc. (We have one but it's disabled on both hosts at a service level)

xnx wrote: »

Sorry, i'm not gonna lie I didn't read all your post :P

Haha, yeah, they're a bit tl;dr, I admit, but they do contain quite a bit of info! :P

atorven

@networker050184 - Good question made me re-think. Since all the frames already have a tag on them a native vlan doesn't need to be specified.
@Genji - I'm just thinking, if the switches can all ping each other fine, try putting your laptops back on the live network and try pinging them and see what happens. Also, when you connect both machines to the same switch do you still have the same issue?

santaowns

Do the laptops have any other kinds of networking software? I have seen VPN software block ports also. Heck its windows so you never know it could still be using firewall rules that you don't know of. The thing we can confirm though is that the issue for the laptops being able to ping is between laptop hardware and the laptop software if it can outgoing ping switch.

Genji

Thanks for thinking with me here everyone.

Connecting them to the same switch gives me the same result - can ping the switches, but not the laptops.

We have Cisco's AnyConnect VPN software installed for whenever we need to work outside of the office. I've never seen it actively block traffic when it's not in use but to make sure I'll remove it completely and give it another go.
It seems silly to even try these little things but at this point I'm willing to accept anything from "Is it actually plugged in?" to "You're not running IPv6 in an IPv4 environment are you?" (And yes, they're plugged in and no, I'm not trying to run IPv6 in an IPv4 enviro )
The same goes for plugging them into our production environment; I'd expect them to work perfectly but I'll doublecheck (LT1 is actually my own corp. Laptop from which I'm typing this message).

What I'm currently discussing internally is whether disabling the Windows Firewall service really disables it, a good pointer here is that disabling the firewall immediately boots me off the domain network however I wonder if it applies a basic "BLOCK EVERYTHING" rule set before it disables...

TBev0

The other really lame thing that you can check which has caught me out before is the Windows Network profile. You usually get three options: Home, Work or Public. I'm not sure whether the network profile operates independently to the windows firewall, but best to make sure that both laptops have the same Network profile specified, I'd set both to either home or work as I'd imagine the public one would do some sort of filtering. If anything, one more thing to tick off the list.

Genji

Haha, I got caught out on the network profile thing before. Makes you really scratch your head and wonder why that would make any difference!

Anywho, it works!
THIS: "I wonder if it applies a basic "BLOCK EVERYTHING" rule set before it disables..."

This is exactly what it does. You turn off the service and in a last ditch attempt to still be able to protect you, the firewall draws it's last breath and basically blocks absolutely everything.
Disabling the service and rebooting both machines (Why oh why didn't I do this sooner?) kills off the FW completely and lets me ping.

Thanks to all for the pointers and thoughts.

Now to remove all the interface vlan IPs I set for TSing and continue down the path of getting these boxes configged and shipped.

santaowns

Glad I could assist.

networker050184

99% of the time it's the PC firewall every time!

santaowns

It was indicted that network portion was fine when pc could ping switch.