CEF src ip address behavior

EdTheLad

Topology
http://imageshack.com/a/img674/6375/TEyY0l.jpg

ipv4 addressing uses router numbers in third octet, i.e. link between R1 and R2 is 10.0.12.0/24; between R2 and R6 10.0.26.0/24
All interfaces are running ospf in area 0; each router has lo0 i.e. 1.1.1.1/32 for R1
LDP is enabled on all interfaces and all neighbor sessions are up.


Am i missing something basic here? cef chooses fa0/1 as its egress interface for the flow to 7.7.7.7; I would have assumed it would have used the ip address on fa0/1 as its src ip, but it's actually using the ip from fa0/0. I reran the test without ldp and the behavior is the same.



R1#sh ip route 7.7.7.7
Routing entry for 7.7.7.7/32
Known via "ospf 1", distance 110, metric 31, type intra area
Last update from 10.0.15.5 on FastEthernet0/1, 01:40:42 ago
Routing Descriptor Blocks:
10.0.15.5, from 7.7.7.7, 01:40:42 ago, via FastEthernet0/1
Route metric is 31, traffic share count is 1
* 10.0.12.2, from 7.7.7.7, 01:40:42 ago, via FastEthernet0/0
Route metric is 31, traffic share count is 1

From the routing table above you would expect the flow to go via 10.0.12.2 on Fa0/0 due to the "*"

R1#sh ip cef 7.7.7.7
7.7.7.7/32, version 23, epoch 0, per-destination sharing
0 packets, 0 bytes
tag information set
local tag: 21
via 10.0.15.5, FastEthernet0/1, 0 dependencies
traffic share 1
next hop 10.0.15.5, FastEthernet0/1
valid adjacency
tag rewrite with Fa0/1, 10.0.15.5, tags imposed: {18}
via 10.0.12.2, FastEthernet0/0, 0 dependencies
traffic share 1
next hop 10.0.12.2, FastEthernet0/0
valid adjacency
tag rewrite with Fa0/0, 10.0.12.2, tags imposed: {24}
0 packets, 0 bytes switched through the prefix
tmstats: external 0 packets, 0 bytes
internal 0 packets, 0 bytes

Now i'll check the cef exact-path using both possible src addresses from int fa0/0 and fa0/1

R1#sh ip cef exact-route 10.0.12.1 7.7.7.7
10.0.12.1 -> 7.7.7.7 : FastEthernet0/1 (next hop 10.0.15.5)
R1#sh ip cef exact-route 10.0.15.1 7.7.7.7
10.0.15.1 -> 7.7.7.7 : FastEthernet0/1 (next hop 10.0.15.5)

Capture
http://imageshack.com/a/img911/3658/ypC6D7.png

deth1k

Please share what is your understanding of CEF load balancing, I mean what is your expected output? From your output you haven't generated enough src-dst pairs to see how traffic is going to be split hence same next hop is being used.

P.S * (asterisk) actually "rotates" between best paths each time flow is serviced however you will not see that in CLI output

EdTheLad

I'm not testing load-balancing distribution here!

I expect since i'm using destination based cef, my flow will use one of the links as i see above it uses fa0/1. After it chooses a link i expect it to send a packet using the src ip address of the egress interface, above cef is choosing fa0/1 as its egress port and sourcing the locally generated packet from fa0/0 interface ip.

deth1k

Sorry, don't understand your question. From your output router choses same next hop / path regardless of source interface being used so source IP would be whatever you've specified in the ping.

EdTheLad

I'll make it simple for you.
On R1, i type ping 7.7.7.7 without specifying a src ip address. What src ip address will be used for the ping?

deth1k

That made it a whole lot of easy Now i see what you mean. Based on cef exact route command fa0/1 is used hence whatever IP of that interface is would be the source. Does output always look the same when you run "exact" command? It could "flick" over when you did your PING

EdTheLad

It's always the same, my capture is done on fa0/1, packets are being sent out fa0/1 with src ip 10.0.12.1; strange, i wonder is it a bug or just a normal occurrence with equal cost paths i've just never noticed. I'll have to check another ios version, i'm using C3725-ADVENTERPRISEK9-M.
Just wondering what is used to select the src address and if the * has anything to do with it.

deth1k

if you can, try with IOL or IOSv. Also you may try shutting down fa0/0, leave it down for a bit and bring back up to see if path via fa0/1 marked with * (asterisk) and try your capture again

p.s asterisk explanation: https://books.google.co.uk/books?id=-8RA47mPxZgC&pg=PA44&lpg=PA44&dq=cisco+routing+table+asterisk&source=bl&ots=7ndPheqIYe&sig=ASQPtzLFB8KAokKNKxadURnh2ng&hl=en&sa=X&ei=OSiIVOeKEob4UL-fgrgH&ved=0CCkQ6AEwAjgK#v=onepage&q=cisco%20routing%20table%20asterisk&f=false

EdTheLad

Brain fart, locally generated packets are process switched, so shouldn't be any cef here.
The asterisks is only used for process switched traffic, it should actually be using both links and hence the asterisks rotating back and forth.
Still strange behavior going on, but i'll have to put it down to either gns or ios code.

fredrikjj

EdTheLad wrote: »

Brain fart, locally generated packets are process switched, so shouldn't be any cef here.
The asterisks is only used for process switched traffic, it should actually be using both links and hence the asterisks rotating back and forth.
Still strange behavior going on, but i'll have to put it down to either gns or ios code.

I don't know if it's true that a packet doesn't follow cef's rules when it's locally generated. For example, try sending a series of icmp echoes to a destination that has multiple outgoing interfaces and look at what the next hop is. It will be the same for every packet. Then go to each outgoing interface and set ip load-sharing per-packet. Now resend the ping and the next hop will alternate per packet. ip load-sharing per-packet is a CEF command. What does that mean? I don't know exactly.

An explanation that is conceptually consistent with what you are seeing could something like this:

1. A process on the device generates your ICMP packet. It knows the destination because you've specified it using the ping command. You haven't specified a source address. When there are multiple routes to the destination it picks a source interface/ip address according to some rule.
2. The packet is sent to some kind of forwarding process that looks at this packet and picks an outgoing interface according to what CEF says. The outgoing interface picked could be an interface that has a different address configured than what the "packet generation" process picked as the source when it generated your packet. In other words, the source address is generated at an earlier stage and independent of the forwarding decision. The forwarding process doesn't care that the source IP address on the packet belongs to a different interface on the router than what it has chosen as the outgoing interface.

This is pure speculation.

deth1k

load-sharing per packet actually disables CEF

d4nz1g

R1#sh ip cef 7.7.7.7
7.7.7.7/32, version 23, epoch 0, per-destination sharing

"Because per-destination load balancing depends on the statistical distribution of traffic, load sharing becomes more effective as the number of source-destination pairs increase"

Have you considered this?

EdTheLad

@deth1k, CEF supports both per destination and per packet load-sharing.
"ip load-sharing per-packet"

@d4nz1g
Yes, that's regarding how flows are statistically multiplexed over the equal cost paths. Nothing to do with this.

This issue has nothing to do with CEF, locally generated traffic is process switched. There was a bug open against 12.2(1 code for this issue.
I've seen this behavior on both ios and iou 12.4 code, would have issues if rpf was enabled.

Still looking at the behavior i'll update the thread with final conclusions.

d4nz1g

Try on CSR1000v too.

I will observe it later, also. This issue is very intriguing.

EdTheLad

Update:

From what i've seen, the first route learned of a multipath route will get the asterisk next to it, this asterisk doesn't move between paths, it always stays on the first learned route. All locally generated traffic forwarding to a multipath destination will use the interface with the asterisk at the source ip address for the packets. The router will then calculate using a hash function which of the multipath egress interfaces to use, this can be different than the src address. This is either a bug or an oversight in my opinion, both processes of picking the src address and the hash function to find exit interface should be tied together. rfc 2991 explains about how the exit interface is chosen.

Regardless of CEF, fast switching or process switching, a cli generated ping will choose one egress interface and send packets on that interface only. Downstream routers receiving the packets will perform either destination based or per packet load balancing depending on the interface configuration. If the ingress port is CEF enabled, it will be CEF switched, if egress interface is Process switched it will be process switched, any other combination of ingress/egress switching will result in fast switching.

On downstream routers that are configured for per packet switching, i observed the packets alternating between egress ports, but i never see that asterisk move. So i guess the documentation is wrong, it just indicates which interface is used as the source ip of the egress packet!
With a 3 link bundle, when i fail the asterisk interface, the asterisk moves to a new port, the ip address of the asterisk port is again used as the src ip.