Weird loop issue

kpjungle

Hi,

I really hope someone can help me with this puzzle. I have 3 routers hooked up, and performing redistribution. But im having a weird issue with Router1 being able to ping a loopback interface on Router3, going through Router2, but Router2 cant get to the loopback!?

For some reason, if i turn off ip cef, ill be able to ping the loopback interface from router 2 (but then i wont be able to do it from router 1), but how can router 1 relay through router 2 without router 2 being able to do ping it itself?

I have attached a diagram of my topology, along with routing tables for each router.




Router 1:

C    192.168.12.0/24 is directly connected, FastEthernet1/0
     172.16.0.0/24 is subnetted, 1 subnets
O E2    172.16.23.0 [110/20] via 192.168.12.2, 01:04:40, FastEthernet1/0
     9.0.0.0/24 is subnetted, 1 subnets
O E2    9.9.9.0 [110/20] via 192.168.12.2, 01:04:22, FastEthernet1/0
     10.0.0.0/24 is subnetted, 1 subnets
C       10.0.12.0 is directly connected, Serial0/0
R1#ping 9.9.9.9

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 9.9.9.9, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 12/48/132 ms
R1#

Router 2:

C    192.168.12.0/24 is directly connected, FastEthernet1/0
     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.23.0 is directly connected, FastEthernet2/0
     9.0.0.0/24 is subnetted, 1 subnets
R       9.9.9.0 [120/1] via 172.16.23.3, 00:00:11, FastEthernet2/0
                [120/1] via 10.0.12.1, 00:00:10, Serial0/0
     10.0.0.0/24 is subnetted, 1 subnets
C       10.0.12.0 is directly connected, Serial0/0
R2#ping 9.9.9.9

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 9.9.9.9, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
R2#

Router 3:

R    192.168.12.0/24 [120/1] via 172.16.23.2, 00:00:22, FastEthernet1/0
     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.23.0 is directly connected, FastEthernet1/0
     9.0.0.0/24 is subnetted, 1 subnets
C       9.9.9.0 is directly connected, Loopback0
     10.0.0.0/24 is subnetted, 1 subnets
R       10.0.12.0 [120/1] via 172.16.23.2, 00:00:22, FastEthernet1/0
R3#

According to what i can see from router 2's routing table, it has two paths, one of them creating a loop, but again, the question is, how is router 1 able to get through to the loopback?

Thanks alot! this is really puzzling

bighornsheep

According to your diagram, R1 & R2 are running both RIP and OSPF. Thus R1, R2 & R3 are all in the RIP domain. R1 will learn the loopback address from R3 through RIP advertisements. R2 is redistributing RIP into OSPF, so the loopback address becomes an external route advertisement and gets sent to R1 and R3, R3 will ignore that external route because it has a connected route to 9.9.9.9 and the AD is better than OSPF E2 route, R1 will accept the external route because AD is better than the RIP.

I don't know how R1 will be able to still use the RIP route for the ping, but I know the above explains why R1 will only show the OSPF route for 9.9.9.9, try to issue a show ip rip database, you should see the 9.9.9.9 route as a RIP message from R3, I'm thinking this has something to do with CEF? R1 knows to go through R2 in the RIP domain to get to 9.9.9.9?

kpjungle

bighornsheep wrote:

According to your diagram, R1 & R2 are running both RIP and OSPF. Thus R1, R2 & R3 are all in the RIP domain. R1 will learn the loopback address from R3 through RIP advertisements. R2 is redistributing RIP into OSPF, so the loopback address becomes an external route advertisement and gets sent to R1 and R3, R3 will ignore that external route because it has a connected route to 9.9.9.9 and the AD is better than OSPF E2 route, R1 will accept the external route because AD is better than the RIP.

I don't know how R1 will be able to still use the RIP route for the ping, but I know the above explains why R1 will only show the OSPF route for 9.9.9.9, try to issue a show ip rip database, you should see the 9.9.9.9 route as a RIP message from R3, I'm thinking this has something to do with CEF? R1 knows to go through R2 in the RIP domain to get to 9.9.9.9?

Yep, i can account for the distribution as well, just not the ping

bighornsheep

I was just reading my notes on IP Routing, and the CEF section. This got me really interested now...can you post the "show ip cef" and "show ip rip data" from R1? Let's confirm R1 does indeed know 9.9.9.9 via RIP messages, then CEF might give us some hints....I'm thinking R1 is running CEF in per destination mode.......

kpjungle

bighornsheep wrote:

I was just reading my notes on IP Routing, and the CEF section. This got me really interested now...can you post the "show ip cef" and "show ip rip data" from R1? Let's confirm R1 does indeed know 9.9.9.9 via RIP messages, then CEF might give us some hints....I'm thinking R1 is running CEF in per destination mode.......

R1 knows 9.9.9.9 through OSPF because of the admin distance of OSPF, and R2 knows it from two different RIP sources, R1 (because of redistribution back into RIP) and from the "real" source, R3, because of normal RIP procedures.

Of R2:
R2#sh ip cef
Prefix Next Hop Interface
0.0.0.0/0 drop Null0 (default route handler entry)
0.0.0.0/32 receive
9.9.9.0/24 172.16.23.3 FastEthernet2/0
10.0.12.1 Serial0/0
10.0.12.0/24 attached Serial0/0
10.0.12.0/32 receive
10.0.12.2/32 receive
10.0.12.255/32 receive
172.16.23.0/24 attached FastEthernet2/0
172.16.23.0/32 receive
172.16.23.2/32 receive
172.16.23.3/32 172.16.23.3 FastEthernet2/0
172.16.23.255/32 receive
192.168.12.0/24 attached FastEthernet1/0
192.168.12.0/32 receive
192.168.12.1/32 192.168.12.1 FastEthernet1/0
192.168.12.2/32 receive
192.168.12.255/32 receive
224.0.0.0/4 drop
224.0.0.0/24 receive
255.255.255.255/32 receive
R2#

That again shows the two next-hops (172.16.23.3 and 10.0.12.1)...
still no real "reason" behind it.

bighornsheep

Issue "show ip cef" on R1. I think it will show Se0/0 using R2 as next-hop which is the route learnt from RIP.....

On R2, you should be able to ping 9.9.9.9, cef shows that it will go thru Fe using R3 as the next-hop.

btw, we're using real gear here right? Just to eliminate the possibility of simulation bug.....

kpjungle

bighornsheep wrote:

Issue "show ip cef" on R1. I think it will show Se0/0 using R2 as next-hop which is the route learnt from RIP.....

On R2, you should be able to ping 9.9.9.9, cef shows that it will go thru Fe using R3 as the next-hop.

btw, we're using real gear here right? Just to eliminate the possibility of simulation bug.....

Its correct on R1 (R1 can easily ping 9.9.9.9). On R2, R2 cannot ping 9.9.9.9.

Its dynamips, so not real gear, but real IOS.

Netstudent

You should filter your redistribution and not give it free rein, especially when doing mutual redistribution of a classless to a classful routing protocol.

When reditributing OSPF into RIP on R1, create a route-map or distribute-list that blocks the route to 9.9.9.0 from being redistributed back into RIP. This will fix the loop and let everyone know of 9.9.9.0

By looking at the routing table in router2, one would expect to at least get a ping reply every other ping request due to RIP load balancing over the incorrect path.

But since you are having different results with CEF, you could have a L3 and L2 issue with suboptimal routing and with the segment between R2 and R3. Or possibly a L2 bug in the hypervisor. I would start by fixing the route.

kpjungle

Netstudent wrote:

You should filter your redistribution and not give it free rein, especially when doing mutual redistribution of a classless to a classful routing protocol.

When reditributing OSPF into RIP on R1, create a route-map or distribute-list that blocks the route to 9.9.9.0 from being redistributed back into RIP. This will fix the loop and let everyone know of 9.9.9.0

By looking at the routing table in router2, one would expect to at least get a ping reply every other ping request due to RIP load balancing over the incorrect path.

But since you are having different results with CEF, you could have a L3 and L2 issue with suboptimal routing and with the segment between R2 and R3.

This is never something I would do on purpose, this is a CCIE lab I found, and thought it was good practice on figuring out how to troubleshoot stuff and maybe learn something. I would deffinately make a route-map, or at least redistribute with a larger metric to avoid the loop. But its puzzling how one router can relay to the destination, whereas a local packet cannot reach the destination. Thats what gets to me

networker050184

Have you done a debug ip packet on the routers to see where it is failing?

It may not be that router 2 can not get to the destination it may be the destination can not get back....

WRKNonCCNP

like bighornsheep said, it looks like you are using per-destination mode cef. if i understand this correctly, R2 will alternate between next hops (L2 adjacencies, in this case R1 and R3, since both are next hops to reach 9.9.9.0).
again, if i read correctly from cisco's website, per-destination mode cef chooses the next hop from its adjacency table based on a hash made out of the source and destination IP address of the incoming packet (http://www.cisco.com/en/US/products/hw/modules/ps2033/prod_technical_reference09186a00800afeb7.html).

If that is the case, when pinging from R1 to 9.9.9.9, the source is 192.168.12.1 and that source/dest IP address combo is switched using a certain adjacency, in this case 172.16.23.3. When pinging from R2, the 172.16.23.2/9.9.9.9 source/dest combo is used, but since cef is in per-destination mode, the next available adjacency is used to reach 9.9.9.0 network, which is 10.0.12.1. Therefore, the pings from R2 just loop between R2 and R1, where pings from R1 to 9.9.9.9 work successfully.

I am not positive about this, but it makes sense to me. I just thought i would throw it out there. Please correct me if anyone thinks i am wrong.

tech-airman

kpjungle,

Questions:

1. R1
   [list=1:97d5751393]
2. What are the configurations for the routing protocols configuration?
3. What are the configurations for redistribution?

[*]R2

1. What are the configurations for the routing protocols configuration?
2. What are the configurations for redistribution?

[*]R3

1. What are the configurations for the routing protocol configuration?
2. What are the configurations for redistribution?

[/list:o:97d5751393]

kpjungle

tech-airman wrote:

kpjungle,

Questions:

1. R1
   [list=1:69269b187b]
2. What are the configurations for the routing protocols configuration?
3. What are the configurations for redistribution?

[*]R2

1. What are the configurations for the routing protocols configuration?
2. What are the configurations for redistribution?

[*]R3

1. What are the configurations for the routing protocol configuration?
2. What are the configurations for redistribution?

[/list:o:69269b187b]

First off, another weird behavior.. I started up my dynamips box to get the info, and with ip cef enabled on all devices, now I can ping from R2 to R3 (loop), but not from R1 to R3 (loop), even though i can from R1 ping to R2 and its route states the next hop as R2, and i can ping that interface. With ip cef disabled on R2, it all works again.

Router 1:
router ospf 1
log-adjacency-changes
redistribute rip subnets
network 192.168.12.1 0.0.0.0 area 0
!
router rip
version 2
redistribute ospf 1 metric 1
network 10.0.0.0
no auto-summary


Router 2:
router ospf 1
log-adjacency-changes
redistribute rip subnets
network 192.168.12.2 0.0.0.0 area 0
!
router rip
version 2
redistribute ospf 1 metric 1
network 10.0.0.0
network 172.16.0.0
no auto-summary


Router 3:
router rip
version 2
network 9.0.0.0
network 172.16.0.0
no auto-summary


With regards to debugs, yes, heavily, but I lack a debug command on R2 which will allow me to see the packets comming in from R1 -> R3, debug ip packet only shows incomming and outgoing, not "relayed" packets.

I dont know much about CEF, but i would think that even with CEF, when my routing table says on R2:
Routing Descriptor Blocks:
* 172.16.23.3, from 172.16.23.3, 00:00:10 ago, via FastEthernet2/0
Route metric is 1, traffic share count is 1
10.0.12.1, from 10.0.12.1, 00:00:07 ago, via Serial0/0
Route metric is 1, traffic share count is 1

The little star would point out the path next choosen, at least for the next packet if not the whole "session". This is weird

Thanks for the help everyone though.

kryolla

to show relayed packets you need to turn cef off on both interfaces. What I dont understand is how is R2 getting the 9. network from R1 via the serial interface with the same hop count when R1 doesnt have any RIP routes in the routing table. Shut down the serial int and see what happens

kpjungle

kryolla wrote:

to show relayed packets you need to turn cef off on both interfaces. What I dont understand is how is R2 getting the 9. network from R1 via the serial interface with the same hop count when R1 doesnt have any RIP routes in the routing table. Shut down the serial int and see what happens

They are being redistributed from OSPF back into RIP on R1, also with a metric of 1. If i was to prevent the loop, i could redistribute it with a metric of 2 or more.

bighornsheep

Okay, I couldn't resist but to lab this up myself. As tired as I've been coming home from work and as old as my gear, I've connected as per your diagram and I figure the following:

R1 - 2503, running RIP, OSPF, redistribute OSPF into RIP, redistribute RIP into OSPF
R2 - 2514, running RIP, OSPF, redistribute OSPF into RIP, redistribute RIP into OSPF
R3 - 2503, running RIP

R1:

ip cef
!
interface Ethernet0
 ip address 192.168.12.1 255.255.255.0
!
interface Serial0
 ip address 10.0.12.1 255.255.255.0
 clockrate 64000
!
router ospf 1
 log-adjacency-changes
 redistribute rip subnets
 network 192.168.12.0 0.0.0.255 area 0
!
router rip
 version 2
 redistribute connected
 redistribute ospf 1
 network 10.0.0.0

R2:

ip cef
!
interface Ethernet0
 ip address 192.168.12.2 255.255.255.0
!
interface Ethernet1
 ip address 172.16.23.2 255.255.255.0
!
interface Serial0
 ip address 10.0.12.2 255.255.255.0
!
router ospf 1
 log-adjacency-changes
 redistribute rip subnets
 network 192.168.12.0 0.0.0.255 area 0
!
router rip
 version 2
 redistribute connected
 redistribute ospf 1
 network 10.0.0.0
 network 172.16.0.0
!

R3:

ip cef
!
interface Loopback1
 ip address 9.9.9.9 255.255.255.255
!
interface Ethernet0
 ip address 172.16.23.3 255.255.255.0
!
router rip
 version 2
 network 9.0.0.0
 network 172.16.0.0
!

As for why R1 can reach 9.9.9.9/32 but not R2, it is indeed because of cef per-destination switching. R1 pings 9.9.9.9 and hands off packet to R2, and for that source-destination pair, R2 uses the first route [120/1] via 172.16.23.3, 00:00:11, FastEthernet2/0, packet gets to R3, and R3 returns echo-reply to R1 using route R 192.168.12.0/24 [120/1] (though the return path shouldn't have been this case if you routed as per your topology, you would have had to redistributed connected routes into RIP for this to work as I have done above.) But, when R2 sends ping to 9.9.9.9, CEF per-destination chooses the second route for the new source-destination pair, and forwards it to R1 which sends it back to R2, thus there's a forwarding loop.

See the routing table below:
R1:

Gateway of last resort is not set

C    192.168.12.0/24 is directly connected, Ethernet0
     172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
O E2    172.16.23.0/24 [110/20] via 192.168.12.2, 00:02:45, Ethernet0
O E2    172.16.0.0/16 [110/20] via 192.168.12.2, 00:02:45, Ethernet0
O E2 9.0.0.0/8 [110/20] via 192.168.12.2, 00:02:45, Ethernet0
     10.0.0.0/24 is subnetted, 1 subnets
C       10.0.12.0 is directly connected, Serial0
R1#

R2:

Gateway of last resort is not set

C    192.168.12.0/24 is directly connected, Ethernet0
     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.23.0 is directly connected, Ethernet1
R    9.0.0.0/8 [120/1] via 172.16.23.3, 00:00:22, Ethernet1
     10.0.0.0/24 is subnetted, 1 subnets
C       10.0.12.0 is directly connected, Serial0
R2#

R3:

Gateway of last resort is not set

R    192.168.12.0/24 [120/1] via 172.16.23.2, 00:00:12, Ethernet0
     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.23.0 is directly connected, Ethernet0
     9.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C       9.9.9.9/32 is directly connected, Loopback1
R       9.0.0.0/8 [120/2] via 172.16.23.2, 00:00:12, Ethernet0
R    10.0.0.0/8 [120/1] via 172.16.23.2, 00:00:12, Ethernet0
R3#

All routers can reach all addresses, no forwarding loop.

kryolla

kpjungle wrote:

kryolla wrote:

to show relayed packets you need to turn cef off on both interfaces. What I dont understand is how is R2 getting the 9. network from R1 via the serial interface with the same hop count when R1 doesnt have any RIP routes in the routing table. Shut down the serial int and see what happens

They are being redistributed from OSPF back into RIP on R1, also with a metric of 1. If i was to prevent the loop, i could redistribute it with a metric of 2 or more.

Thanks for the clarification but if you knew you created a loop how come you didn't correct it? If it was corrected then R2 will only have one route to 9. network or were you trying to figure out why when you turn off cef you can ping the 9. network.

This doesn't pertain to this thread but in a production network do you guys really have 2 paths to the same destination with CEF load balancing going on in your wan links or do you use MLP for load balancing? At my work we don't use load balancing but use the second path as backup.

This reminds me I need to do more redistributing scenarios but I think this one was a nice intro to 2 point of redistribution and you can just add to this to make it more complex.

Also debug ip packet only works for process switch packets

kpjungle

kryolla wrote:

kpjungle wrote:

kryolla wrote:

to show relayed packets you need to turn cef off on both interfaces. What I dont understand is how is R2 getting the 9. network from R1 via the serial interface with the same hop count when R1 doesnt have any RIP routes in the routing table. Shut down the serial int and see what happens

They are being redistributed from OSPF back into RIP on R1, also with a metric of 1. If i was to prevent the loop, i could redistribute it with a metric of 2 or more.

Thanks for the clarification but if you knew you created a loop how come you didn't correct it? If it was corrected then R2 will only have one route to 9. network or were you trying to figure out why when you turn off cef you can ping the 9. network.

This doesn't pertain to this thread but in a production network do you guys really have 2 paths to the same destination with CEF load balancing going on in your wan links or do you use MLP for load balancing? At my work we don't use load balancing but use the second path as backup.

This reminds me I need to do more redistributing scenarios but I think this one was a nice intro to 2 point of redistribution and you can just add to this to make it more complex.

Also debug ip packet only works for process switch packets

I did correct it, but i wanted to explore the reasoning behind this behavior, in order to troubleshoot an unknown network in the future. If I had encountered this in a production network, and I had just issued a ping from R1 to 9.9.9.9, I might have thought everything was just great, with full reachability, but as shown, that would be a false sense of security.

But thanks everyone for all the help. I know i learned alot from this