Strange static routing issue

rakem

I have the following configuration on some stacked (3 of them) Cisco 3750s.

ip route 10.179.0.0 255.255.0.0 10.191.255.5
ip route 10.179.0.0 255.255.0.0 10.191.255.9 50
ip route 10.191.0.0 255.255.0.0 10.191.255.5
ip route 10.191.0.0 255.255.0.0 10.191.255.9 50

So here you can see we have multiple static routes to the same destinations, and floating statics to an alternate path.

Here is the relevant interface config;

interface GigabitEthernet3/0/25
 no switchport
 ip address 10.191.255.6 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 no ip mroute-cache
 load-interval 60
end

and

interface GigabitEthernet3/0/26
 no switchport
 ip address 10.191.255.10 255.255.255.252
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 no ip mroute-cache
 load-interval 60
end

So in the static routes we are using interface Gi3/0/25 as the primary one and Gi3/0/26 as the backup.

Here is the relevant part of the route table when everything is working ok.

S       10.191.0.0/16 [1/0] via 10.191.255.5
S       10.179.0.0/16 [1/0] via 10.191.255.5

Now, when I shut down Gi3/0/25 look what happens;

S       10.191.0.0/16 [50/0] via 10.191.255.9
S       10.179.0.0/16 [1/0] via 10.191.255.5

The 10.179.0.0/16 network keeps 10.191.255.5 as the next hop in the routing table, even though it is now down. The other route changes as expected and I can still ping devices in the 10.179.0.0 network, and a trace route shows it is using the path through 10.191.255.9.

Switch#ping 10.179.33.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.179.33.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms
Switch#trace 10.179.33.1

Type escape sequence to abort.
Tracing the route to 10.179.33.1

  1 10.191.255.9 0 msec 0 msec 0 msec
  2  *  *  *
  3  *  *  *

(Don't worry about the 2nd and 3rd hop time-outs)

So why is the route table using the floating static route to the 10.191.0.0/16 network but not the 10.179.0.0/16 network. As shown i can still ping to devices in 10.179.0.0/16 and a trace shows it uses the right first hop. Seems a bit weird to me...

Thoughts?

sifresiuydurma

Hi,

ip route 10.179.0.0 255.255.0.0 10.191.255.5

and

interface GigabitEthernet3/0/25
no switchport
ip address 10.191.255.6 255.255.255.252

chmorin

I'm not sure what the post above me was trying to point out.

This is a strange issue. You shouldn't have to, but try flushing out the routing table and see if it shows up properly. The router is acting the correct way, only it looks like the routing table didn't keep up.

That is all I got... by my accounts your configurations are proper. Then again, I need to start studying for my BSCI again.

trackit

sifresiuydurma wrote: »

Hi,

ip route 10.179.0.0 255.255.0.0 10.191.255.5

and

interface GigabitEthernet3/0/25
no switchport
ip address 10.191.255.6 255.255.255.252

no, i guess its supposed to be that way... 10.191.255.6 is the local address on switch and 10.191.255.5 is the remote "next hop" address...

chmorin

trackit wrote: »

no, i guess its supposed to be that way... 10.191.255.6 is the local address on switch and 10.191.255.5 is the remote "next hop" address...

Yeah, you wouldn't make a static route on yourself to yourself. Haha.

networker050184

Your config looks good, so not sure why that one is not changing. Have you tried a "debug ip routing static detail" to see if you get anything useful from that? Since it looks to be taking the right path you might try a "clear ip route *" to see if that gets it going like chmorin said. You really shouldn't need to do that though. Might be some kind of IOS bug so you should look into that.

One other thing to try is to use the track feature to see if that will make it act right.

APA

IOS recursively resolves static-next hops that are explicitly assigned by IP address. For both routes to get withdrawn upon Gi3/0/25 going down you could as networker said track the interface and cause an event drive route withdrawal or configured the static routes with a next hop of a physical interface (BAD!!!! - unless its a point to point interface then not so bad....)

The reason it doesn't change the route to 10.179.0.0/16 is because the 10.191.0.0/16 route comes back up via 255.9 and therefore makes 10.191.255.5 (pseudo reachable) again because it has a route to it even though the phyical interface is down.

Recursive behaviour will see the following ---- route 10.191.255.5 via the best path available which is the static route pointing 10.191.0.0 to next-hop 10.191.255.9. Hence why you still get successful connectivity.

IOS doesn't strictly stick to next-hops being directly connected for static routes..... it just recursively resolves them and only when it has no route to the next-hop available does it withdraw the route from the RIB.

Hope this explains the situation... as it definitely isn't a bug.

If you need the RIB to change.... may I suggest networkers tracking method above... or a dynamic solution?

networker050184

Good catch APA! I didn't even pay attention to the fact that they static route encompassed the interface range. At least I was right about the debug would have shown you something!

Cyanic

Thanks APA. It makes sense now.

rakem

APA wrote: »

The reason it doesn't change the route to 10.179.0.0/16 is because the 10.191.0.0/16 route comes back up via 255.9 and therefore makes 10.191.255.5 (pseudo reachable) again because it has a route to it even though the phyical interface is down.

Recursive behaviour will see the following ---- route 10.191.255.5 via the best path available which is the static route pointing 10.191.0.0 to next-hop 10.191.255.9. Hence why you still get successful connectivity.

If you need the RIB to change.... may I suggest networkers tracking method above... or a dynamic solution?

Thanks for the reply.... not sure i underatand 100% what is happening, how does the router know the 10.179.0.0/16 network is reachable via 10.191.255.9 when 10.191.255.5 is down?

I did a debug ip route static detailed, shut the interface down and got this;


Apr 23 06:39:35: IP-Static: 10.179.0.0 255.255.0.0 10.191.255.5 Path = 2 5 6
Apr 23 06:39:35: IP-Static: 10.179.0.0 255.255.0.0 10.191.255.5 Path = 2 5 6 8, route table deleted, recursive flag set
Apr 23 06:39:35: IP-Static: 10.179.0.0 255.255.0.0 10.191.255.9 Path = 2 3 5 7, route table updated, recursive flag clear 8, route table deleted, recursive flag set
Apr 23 06:39:35: IP-Static: 10.179.0.0 255.255.0.0 10.191.255.9 Path = 2 3 5 7, route table no change, recursive flag clear
Apr 23 06:39:35: IP-Static: 10.191.0.0 255.255.0.0 10.191.255.5 Path = 2 5 6
Apr 23 06:39:35: IP-Static: 10.191.0.0 255.255.0.0 10.191.255.5 Path = 2 5 6 8, route table deleted, recursive flag set
Apr 23 06:39:35: IP-Static: 10.191.0.0 255.255.0.0 10.191.255.9 Path = 2 3 5 7, route table updated, recursive flag clear 8, route table deleted, recursive flag set
Apr 23 06:39:35: IP-Static: 10.191.0.0 255.255.0.0 10.191.255.9 Path = 2 3 5 7, route table no change, recursive flag clear
Apr 23 06:39:35: IP-Static: 10.179.0.0 255.255.0.0 10.191.255.5 Path = 2 3 5 7, route table updated, recursive flag clear
Apr 23 06:39:35: IP-Static: 10.191.0.0 255.255.0.0 10.191.255.5 Path = 2 3 5 6 8, route table deleted, recursive flag clear
Apr 23 06:39:36: %LINK-5-CHANGED: Interface GigabitEthernet3/0/25, changed state to administratively down


then when i bought the interface back up


Apr 23 06:44:18: IP-Static: 10.179.0.0 255.255.0.0 10.191.255.5 Path = 2 3 5 7, route table no change, recursive flag clear
Apr 23 06:44:18: IP-Static: 10.179.0.0 255.255.0.0 10.191.255.9 Path = 2 3 5 7, route table no change, recursive flag clear
Apr 23 06:44:18: IP-Static: 10.191.0.0 255.255.0.0 10.191.255.5 Path = 2 3 5 7, route table updated, recursive flag clear
Apr 23 06:44:18: IP-Static: 10.191.0.0 255.255.0.0 10.191.255.9 Path = 2 3 5 7, route table no change, recursive flag clear


I dont really care that the route table does not show any changes... was more just curious as to why i still had connectivity. I will do some reading.


So does the fact that im routing to 10.191.0.0/16 via an address within that subnet (10.191.255.5) cause this issue?

networker050184

Your next hop of 10.199.255.5 is reachable via the 10.191.0.0/16 static route. So when that one changes you now have a route to your next hop so it doesn't have to change. I would use this.

ip route 10.179.0.0 255.255.0.0 GigabitEthernet3/0/25 10.191.255.5
ip route 10.179.0.0 255.255.0.0 GigabitEthernet3/0/26 10.191.255.9 50
ip route 10.191.0.0 255.255.0.0 GigabitEthernet3/0/25 10.191.255.5
ip route 10.191.0.0 255.255.0.0 GigabitEthernet3/0/26 10.191.255.9 50

We actually ran into a somewhat similar issue with customer static routes and came up with this solution.

rakem

Ok, but the next hop of 10.191.255.5 should be down. If i shut the interface down on one the 10.191.255.6 side, this would make 10.191.255.5 go down, so really the route to 10.191.255.5 is invalid, in my opinion anyway.

This link backs up what you have said.... to use the interface and IP in the ip route command

Specifying a Next Hop IP Address for Static Routes - Cisco Systems

Basically the link above says "You always keep the static route in the routing table as long as you have a route to the next hop"

But the next hop in this case is shutdown. Why is this still considered a valid next hop?

APA

Yep networkers solution is spot on...... I was going to edit my post above but he has beat me to it

Rakem... re-read my post above.... IOS doesn't validate the next-hop... it just recursively resolves it, if it sees that it has a path available then it doesn't withdraw the static route.

So in this case.... 10.191.0.0/16 is available via 10.191.255.9 which makes 10.191.255.5 available via the route mention previous as it's covered under the /16.

Once you bring the physical interface up again.. the /16 is no longer the more specific route to 10.191.255.5 as the /30 on Gi3/0/25 becomes the more specific route and IOS changes thew next-hop for the static route to 10.191.0.0 accordingly.

So you still get connectivity because 10.179.0.0/16 is routed via the best available path to 10.191.255.5 which is via 10.191.255.9 as shown in your traceroutes when the interface is down.

This is the problem with IOS not properly validating static-route next-hops...... in JUNOS by default it expects a directly connected peer... if not you can explicitly tell it that the peer is not directly connected and it will then behave like IOS...

rakem

APA wrote: »

Rakem... re-read my post above.... IOS doesn't validate the next-hop... it just recursively resolves it, if it sees that it has a path available then it doesn't withdraw the static route.

Ok that clears things up. This is interesting, you would think that a router would have enough smarts to know that since one side of a /30 is down, then the other side is down as well, since its a point to point link.

Actually had a demo of JUNOS a few weeks back, some of the features it has makes IOS look like windows 3.1.
I love the commit check features.