NBMA Spoke-to-Spoke Comm not working - Solved

ande0255

I use the Chris Bryant NBMA hub-and-spoke topology to learn most concepts on thus far, and have a few text docs I keep around to for when I want to keep a fresh start, I can just slap them into R1 R2 and R3 to be off and going.

However while labbing some Redistribution earlier I lost all spoke to spoke communication to the hub, however the spokes can ping the hub and likewise, but not each other. I actually even did a 'write erase' and reloaded all 3, applied the below configs which are minimal to have always gotten full connectivity off the bat, but I am still not pinging across that Hub between spokes with absolutely no other configs than seen below.

Anyone every run into a weird glitch like this that bricked a router in their lab? Through pings I can tell physical cabling is fine, can something logical I did be stuck in memory? I have completely shut them down for an hour before I went back it.

Attached is a Topology (DLCI's are different in topolgy cause I am way tired), and below are my starter configs that have always worked to start over quick:

R1:

en
conf t
no ip domain-lookup
hostname R1
enable secret CCNP
line con 0
logg sync
no exec-t
line vty 0 4
password CCNP
login
logg sync
exit
int s0/0
ip add 172.12.123.1 255.255.255.0
encap frame
no frame inver
frame map ip 172.12.123.2 122 broadcast
frame map ip 172.12.123.3 123 broadcast
no shutdown
exit
do wr

R2:

en
conf t
no ip domain-lookup
hostname R2
enable secret CCNP
line con 0
no exec-t
logg sync
line vty 0 4
password CCNP
login
logg sync
exit
int s0/0
ip add 172.12.123.2 255.255.255.0
encap frame
no frame inver
frame map ip 172.12.123.1 221 broadcast
frame map ip 172.12.123.3 321 <- I HAVE FOUND MY OWN UNDOING
no shutdown
exit
do wr

R3:

en
conf t
no ip domain-lookup
hostname R3
enable secret CCNP
line con 0
logg sync
no exec-t
line vty 0 4
password CCNP
login
logg sync
exit
int s0/2
ip add 172.12.123.3 255.255.255.0
encap frame
no frame inver
frame map ip 172.12.123.1 321 broadcast
frame map ip 172.12.123.2 321
no shut
exit
do wr

I confirmed the frame switch is the same config I've used for the last 6 years or so, these configs should work for full connectivity, but for some reason that hub is not allowing pings to pass through (but will advertise routes when I had more routers involved).

This is driving me fcking crazy, it can't be cabling because all serial lines are sending and returning pings between the hub to the individual spoke.

I ran continuous pings on both spokes to the other, and got back on R1, to find this:

R1#debug ip pack
IP packet debugging is on
R1#
ASR#2
[Resuming connection 2 to r2 ... ]

R2#ping
Protocol [ip]:
Target IP address: 172.12.123.3
Repeat count [5]: 10000
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 10000, 100-byte ICMP Echos to 172.12.123.3, timeout is 2 seconds:
.....
ASR#3
[Resuming connection 3 to r3 ... ]

*Mar 1 17:37:36.186: %SYS-5-CONFIG_I: Configured from console by console
R3#ping
Protocol [ip]:
Target IP address: 172.12.123.2
Repeat count [5]: 10000
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 10000, 100-byte ICMP Echos to 172.12.123.2, timeout is 2 seconds:
..
ASR#1
[Resuming connection 1 to r1 ... ]

R1#
R1#
R1#
R1#debug all

R1 isn't seeing anything for traffic between the two spokes, I'll save you the output but if I ping from either spoke to the Hub you will see the output of the RIB and FIB tables from successful pings.

I did the same on a failing ping on a spoke, and the output only referenced the FIB table, never the RIB.

ande0255

I actually caught it when reviewing it again not so tired... R2 has frame maps to both 221 and 321 on it's config, once that changed pings came right back.

The real issue was I got so used to R4 and R5 being off FastEthernet ports on pinging from them to a spoke, I never really ever actually tested pings from the spokes themselves. Additionally route advertisements will still make it to both spokes despite not being able to ping eachother, so IT WILL LOOK LIKE IT CAN PING, BUT IT DEFINITELY CANNOT.

Lesson learned there to re-check configs, I'm fairly certain in my blog thing I didn't think about it, but I generally test pings between a non-spoke router I often use so I've never noticed that error for ages.

Very interesting.