Journal

bermovick

So I'm playing with multicast some more -- simple setup, using PIM-DM so I don't even have to worry about an RP or anything, using the ine tasks but coloring outside the lines a little bit, and I've got something ... weird happening.

The pertinent part of the lab here is R6 as the source (pinging 224.10.10.10), connected via ethernet to R4, which is connected to R5 both via direct serial and frame-relay. On R5 the RPF interface is the direct serial link. R5 also connects down towards a receiver, but the problem is strictly between R4 and R5 here.

The 'task' was to simulate an RPF failure by removing PIM from the direct serial and enabling it on the frame-relay interface itself. Pings fails - turning off mroute-cache and debugging ip mpacket shows the RPF failures on R5. So far so good. The task is to make it work again over the FR. In the solutions they use a static "default" mroute pointing out the FR interface. Makes sense, but here is where I go off-track.

Instead of an mroute, I create a static route (no m) for the source (155.1.146.6/32) with an exit interface of the FR interface. This should work since the static route has lower AD then EIGRP or OSPF, and /32 would be the longest match. R5 now works, but since OSPF is redistributing static, R4 thinks the RPF is the FR back towards R5. A bit of an oops on my part, but that's what learning is. I remove the /32 static route on R5 and put on a /24 on instead. R4 should ignore the redistribution since it's directly connected to 155.1.146.0/24.

Pings still fail. What?

*Mar  1 03:37:48.115: IP(0): s=155.1.146.6 (FastEthernet0/1) d=224.10.10.10 id=301, ttl=254, prot=1, len=114(100), not RPF interface

WHY does R4 still have a /32 route for 155.1.146.6 pointing towards R5? Hmmm, I think -- I would have thought OSPF would have updated it's removal when I removed the static route. Perhaps redistribution isn't quite as fast (seems fishy to me, but who knows, right?). No go - I come back from grabbing something to eat to see the last update of the /32 route 25 minutes ago.

FINE - I clear the ospf process, which doesn't get rid of the /32 route. I bounce the FR interface (and associated OSPF neighborship) and the /32 is gone!

Rack1R4(config-subif)##do sh ip route 155.1.146.6
Rack1R4(config-subif)#

Odd, but at least that rogue route is gone. I start up pings again. Pings fail. RPF failure on R4.

Rack1R4#sh ip route 155.1.146.6
Routing entry for 155.1.146.6/32
  Known via "ospf 1", distance 171, metric 20, type extern 2, forward metric 64
  Last update from 155.1.0.5 on Serial0/0.1, 00:11:01 ago
  Routing Descriptor Blocks:
  * 155.1.0.5, from 150.1.5.5, 00:11:01 ago, via Serial0/0.1
      Route metric is 20, traffic share count is 1

OK FINE. What does the ospf database show then?

Rack1R4#sh ip ospf database external 155.1.146.6

            OSPF Router with ID (150.1.4.4) (Process ID 1)

                Type-5 AS External Link States

  Routing Bit Set on this LSA
  LS age: 905
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 155.1.146.6 (External Network Number )
  Advertising Router: 150.1.5.5
  LS Seq Number: 80000002
  Checksum: 0xD5F2
  Length: 36
  Network Mask: /32
        Metric Type: 2 (Larger than any link state path)
        TOS: 0
        Metric: 20
        Forward Address: 0.0.0.0
        External Route Tag: 0

It's being redistributed by R5? ok, R5 is doing a lot of redistributing, so that makes things a little bit complicated, but where is R5 getting this /32 route from to be redistributing it?

Rack1R5#sh ip route 155.1.146.6
Routing entry for 155.1.146.0/24
  Known via "static", distance 1, metric 0 (connected)
  Redistributing via eigrp 100, ospf 1
  Advertised by eigrp 100
                ospf 1
  Routing Descriptor Blocks:
  * directly connected, via Serial0/0
      Route metric is 0, traffic share count is 1

... I see. It isn't. I'm kindof at a loss here. I'll continue "troubleshooting" (by which I mean look at semi-random things, trying to find pieces that fit, but it's wild guesses at this point). Anyone know what I'm missing?


[UPDATE]
While reading over my post I noticed the # in front of my 'show ip route' that finds the /32 route gone, meaning I didn't actually (successfully) check if it was gone or not. Repeating the task shows it was not, so my original hypothesis (that the introduction of packets from the source created a /32 route) was incorrect. This /32 is being generated as soon as the OSPF peering comes up. Troubleshooting continues...

bermovick

Digging further, I see that OSPF and EIGRP are cross-redistributing each other - neither is redistributing static, so it's not something to do with the static address itself. This leads me to hypothesis #2: That it has to do with the FR multipoint network & OSPF and their odd interactions. GOSH I love frame relay and am sad to see it no longer on the exams (/sarcasm)


[UPDATE]
So, I closed GNS3 and created a new topology, much simpler in nature, to test it. I couldn't get the same failure since I SUCK. I ended up re-loading the original topology, and now it works (not with the /24 route -- that fails further downstream due to EIGRP no longer having the /24 downstream cause of the static route on R5) but with MY ORIGINAL /32 ROUTE THAT SHOULD HAVE WORKED IN THE FIRST PLACE.

GNS3 issue? Multicast being flaky? User Error? Which thing did I learn today? (No really, cause I don't know)

gorebrush

I hate multicast too. But my hatred is only due to a lack of practice.

maxpowers

I have enjoyed reading this! I feel your pain

bermovick

Took the past couple days to go through multicast (PIM) slowly. It's a bit trickier just because of how it depends on the routing protocols in use - unexpected behavior is easy to run into. I managed to make it through fairly well today, and have moved on to MPLS.

MPLS! Task #1 and I'm stuck. Well crap.

[update]
Found where I'd configured something incorrectly. It made sense how I configured it, so I'll have to ponder this some.

I'm wishing about now that I had a dedicated MPLS book.

bermovick

Got through task 1 and then stuck on task 2. So ... yeah.

Trying to see if I can find external information beyond "here's how you set up LDP, and how you set up MPBGP & all the other pieces that make up L3VPNs", since there's no mention of leaking routes between VRFs with static routes (task 1), or LDP authentication, or mass-enabling of LDP (task 2) and I hate just looking at the answers vs researching elsewhere, answering the question, and THEN verifying against the answer

(yes, I know I contradicted a previous post where I consigned myself to doing exactly that)

bermovick

I've spent the weekend refreshing myself on MPLS.

Tuesday marks my "week 8" so I'm supposed to be moving on to workbook 2, but I may give myself some extra time to work on MPLS more. I may move ahead too though, trusting the larger exercises to get everything up to speed and allocate time later for anything that isn't.

bermovick

Struggling through the INE MPLS section today.

One thing I've been noticing that I'm pretty unsure about is what parts of BGP peering gets configured under the global bgp process and what parts get configured under the vpnv4 sub-process? Obviously (well, to me), the things that are needed only for vpnv4 (extended community) would be there, but I found in a lab that next-hop-self had to be under the sub-process when eBGP was used for peering to a customer. Putting it under the global process' neighbor statements was not working.

... At least I think that was the case. Troubleshooting MPLS is difficult because it feels like a house of cards sometimes.


As an update, for my current lab, my route reflector wouldn't receive the vpnv4 route until I added the "route-reflector-client" to the neighbor statement under the vpnv4 sub-proces, rather than under the global bgp process for each neighbor.

bermovick

Week 8. Dang it went by fast. Loading up wb2 and poking through scenario 1. Hit some layer 2 stuff GNS3 can't do (configuring bpduguard: easy enough, so I'm not worried about not being able to configure it. private vlans: errrr. I better set up some switches at work here to practice it).

Managed to draw up a layer 2 diagram using show cdp neighbors but drawing a layer 3 one seems trickier. First off the way I'm thinking would be very time consuming (And I just realized it's similar to how ospf does!), and secondly it would be VERY messy. If anyone is willing to share any tips they've found on drawing these up, I'd appreciate it. Until then I'm going to try it and see just how badly it turns out.

Update:
Well, I guess it could have been worse. Took about 35 minutes to do, and I only see 1 difference comparing what I drew with the layer 3 diagram provided (and that may be a typo in the gns 3 .net file or something purposefully put in the configuration)

Update2 (Since I'm trying to limit myself to 1 post per day).
Today was ... quite humbling. I only managed to get through 3 tasks (each task has 5-8 parts), and at least 1 part in each task I needed help with (although frequently it was just because I wasn't sure exactly what was wanted. Like a riddle, once I saw the solution, it made sense but before then I wasn't certain).

I also had more issues with gns3 - ethernet interfaces that sometimes take nearly a minute to begin passing traffic, and missing/incorrect information in the .net file that required some outside-the-tasks troubleshooting.

bermovick

Plugging along still. Some sections in Vol 2 area a breeze and others I struggle with, but I suppose that's the point, isn't it?

I still have to complain about how annoying some of the wording is. For a section on configuring LDP between 3 of the routers it has you configure 'so labels are only generated for the respective router's loopback0 interfaces'. So of course I configure an ACL on each of the 3 that only includes its Lo0 address and apply it. Learning experience since on the center router it also blocked re-advertisement of the edge routers to the other edge.

So I check the answers to see if I'm on the right track / how they do it (maybe a 3-line ACL for all 3 devices loopbacks?) and I see the ACL mask of 0.0.255.255 - which in no way covers only "the respective router's loopback0 interfaces" (it includes ALL routers in the topology's loopback0 interfaces).


(and in the next task I perfectly set up a nice MPBGP peering, forwarding traffic for 2 different VRFs, to find out that was 100% not what was wanted)

bermovick

Does anyone know of any really good sources that covers the pitfalls and gotchas you can run into dealing with redistribution? I'm trying to get through the example in my current INE lab but I'm just not seeing why they had to raise the OSPF External AD to prevent loops.

gorebrush

The main thing with redistribution to remember is to look at the points of entry into that specific routing protocols scope.

I.e. if you have two routers redistributing into OSPF, from different protocols - this is where you get into the tricks of raising AD etc.

bermovick

Oh, I understand the concept (I think) in principle, but apparently when I actually see it in practice I don't grasp it.

This specific lab has 3 redistribution points between EIGRP and OSPF and for some reason raises OSPFs external AD to 169 and I have no idea why.

OSPF prefers intra- and inter- area routes over external routes already, so raising it does absolutely nothing as far as I can tell.

fredrikjj

What's the topology?

PS.

OSPF prefers intra- and inter- area routes over external routes already, so raising it does absolutely nothing as far as I can tell.

Generally speaking, modifying OSPF external AD is to prevent feedback and/or suboptimal routing when the other routing protocol has higher AD on its internal routes. This would only be redistributing between OSPF and RIP/IS-IS in a simple two routing domain topology, but if you have multiple domains, things could get weird even with EIGRP. For example, I remember this example from one of the CCNP textbooks:



The prefix from the unspecified domain (could be any routing protocol) enters EIGRP and gets AD = 170. The OSPF/EIGRP border routers will prefer the external OSPF route over the "internal" external EIGRP route and redistribute it back into EIGRP. In not much of an artist, but hopefully you get what I'm trying to say in that picture.

bermovick

Pretty basic. You have 4 OSPF devices doing full-mesh FR (p-2-mp ospf network). 1 of those 4 redistributes to RIP (R1), the other 3 redistribute to EIGRP (R2, R3, R4).

The EIGRP domain is basically R2-R3-R5-R4 in a line.

Nothing else in the topology impacts this.


[EDIT IN RESPONSE TO ABOVE POST'S EDIT]

I ... kindof do, and this is the type of thing I really think I need to get down better. Another example is mentioned here:
http://www.cisco.com/c/en/us/support/docs/ip/open-shortest-path-first-ospf/28025-2ospf-redis.html
Where 2 OSPF domains are redistributing between each other at 2 points, and you get a "race" scenario as to whether the inter-area route or the external route gets installed - since both have AD 110 and the 2 domains don't consider the other's type, it comes down to which is received first.

bermovick

Work has been extra quiet lately and I have access to some old esx hosts capable of running CSR1000v, so I've been able to spend a lot of time working through the workbooks again. Things are going better than last time (I took a short break that ended up turning into a longish break). It's like taking a break lets concepts ferment in my subconscious because things I didn't quite get last time I'm finding easier this time.

I'm finding an amusing problem though on days when I manage to spend almost the entire day labbing. I never thought as much as I type I'd end up with typing fatigue, but after about 5 hours I definitely start to feel some tiredness in my fingers


I also just received a T5500 I'd ordered which I've set up ESX on and CSR1000vs as well, so I can lab at home now too. I never really liked using IOU, even though I had NETMAPs set up for different labs.

gorebrush

There is definitely something about IOU that makes me feel uneasy, which is a bit daft as that is essentially what the lab is built on. I've had a very slow week or two before this one, and I didn't even realise it was that bad until after this week. I've gone through entire BGP and MPLS INE videos - that's around 20 odd hours and done the odd bit of reading here and there.

It is strange how some weeks you seem to drag on and others it just flows nicely. I'm hoping for another 10 weeks of that though owing to the fact my lab is imminent.



Good luck with yours and well done on getting some more progress

silver145

Its not just you, i think IOU feels odd in comparison to the CSR or physical kit also, something about it just feels meh......

I was on the cisco 360 course last week and the system/iou they use seems to feel more realistic, might be a mental thing but that's my 2 cents

bermovick

Exactly! I don't know why, but IOU just feels ... klunky, I guess. Like I have to be careful or it'll just break.

I STILL HATE DOC CD. Things I need to know seem to fall into 1 of 3 categories:
1) I know it; at worst I'll have to use ? to see the exact syntax or order of arguments
2) I know enough that liberal use of ? will lead me to the correct answer, even if it takes several minutes to find the correct command & arguments.
3) I have no idea.

For 1 I don't really need the doc cd, and for 2 I just need to get faster. The doc cd is there as an option though, since I at least know the command to use the command reference, but for 3..........

My current example was ospf's discard-route option. I labbed, and failed. tracing showed because a summary put in a default route, and I sat back and went "huh. How can you keep it from doing that"? Poked at Doc CD, then looked at answer. I can't do that in the real exam.


[update]
I grabbed the entire OSPF pdf and with some Ctrl+f searching, I find that discard routes are mostly discussed in the "OSPFv2 Local RIB" section. So there's that.

joelsfood

I have nightmares about category 3 myself. That's actually one of the things I really find the forums here and at ieoc useful for. If someone asks a question I don't know the answer to, I'll try to look up the answer and help them out, as it helps me spend more time in the docs, and hopefully learn more things so that come lab day, I WON'T have to go to the docs if the subject comes up.

bermovick

After 4 days of very busy work setting up an Aruba wireless controller and clearpass device (avoids rant about it being some other contract and not part of mine), I finally get to get back into studying. I'm a few sections into the multicast section of the INE workbook.

After spending all morning reinstalling my CSR devices (premium license expired, and I thought the easiest thing to do would be to reinstall), reconfiguring the virtual console and rebooting devices that come up but refuse to work, I finally get to start labbing after noon.

....... and pretty much immediately hit a large, thick stone wall. I've spent 2 hours spinning in circles on a simple sparse-mode config that refuses to work.

Perhaps a new set of eyes will see something I'm obviously missing.

R4-R5-R8-R10. R8 and R10 are candidate RPs. R5 is mapping agent. R4 is pinging groups that R10 has joined (on non-in-line interfaces). R10 is also winning the RP due to highest address.

Pings don't work. Most of the time. Sometimes they do. There's no rhyme or reason I can figure out.

When they do work, my debug SEEMS to be sending the packet in the wrong direction, towards a section of the network that doesn't even have multicast-routing configured on it. When they fail, the debugs don't even list that interface.

I'll try from my house where I can copy/paste so maybe someone else can explain WTF is going on here. Maybe. Right now I'm pretty pissed off about it.

silver145

When practicing multicast i have noticed behaviors with the IOU and CSR that should NOT occur - Take this into account to save your sanity

reaper81

Don't blame the software until proven otherwise. Many do that mistake when they have a poor understanding of multicast. Not saying you do but always try to find a logical explanation.

Is the routing to the RP correct? Is the routing to the source correct? Do you have failures all the time or only when switching to the SPT?

silver145

Well when i copied the exact config over to physical devices and it worked without a problem, that was me happy

bermovick

When I (to my knowledge) configured the same thing on my home lab (also CSR), it worked like a charm. It's very possible I missed a part of the config at work (repeatedly since I wiped config a time or two to start over) though. I'll have to check come Monday.

silver145

Some of the IOU have known multicast bugs so i would look into the version you are using also.

gorebrush

Yeah fully aware that Multicast can be "problematic" on IOU myself.

Will probably just do some light labbing up on my physical boxes for that (3x1841 and 4 switches)

bermovick

Back at work today. Looks like I missed putting sparse-dense-mode on the interface that had also joined the group address.

I'm also guessing I mis-read the debug outputs. Man, you think you understand something but as soon as something unexpected occurs you realize how wrong you are!

[EDIT]
Allow me to add, deny statements in the ACL called by a send-rp-announce command REALLY REALLY REALLY screw things up. At least in my lab they did. Documentation claims they make the denied group address act as a dense-mode group, but none of my labs did anything of the sort. Instead, pings to those addresses failed entirely, even when the routers were reconfigured to use dense-mode. I'll lab more because I can't imagine my experience of "this option breaks everything" is how it's supposed to be.

bermovick

Has anyone here ever understood the hash value in the ip pim bsr-candidate command? Doc CD is ... less than helpful, as is the INE blog post (or content in the configuration section of the workbook which is almost word for word the same).

The workbook example sets a hash length of 31 for ... some reason. They don't ever explain why they chose 31 instead of 32. Originally I thought they incorrectly chose 31 and I would see the same RP be used for 2 consecutive groups, but my hash length of 32 does that sometimes too!


On another note: Does anyone else have CSR1000v's occasionally become unreachable? My virtual console is still good but occasionally one will stop being able to ping or be pinged. Rebooting helps but that's nearly 20 minutes of downtime while it does that.

[UPDATE]
OK I counted and reloading a single one only takes 5 minutes, which is much better. After rebooting I could compare the differences between hash lengths of 31 and 32. The 32-bit hash length gives a 60/40 split over 5 consecutive groups, while INE's 31 bit actually gives a 20/80 split (using groups 239.1.1.1-239.1.1.5).

joelsfood

This post seeems to describe it a bit better than the INE blog. To be honest, not sure I've completely wrapped my head around it even with this one though.

https://aitaseller.wordpress.com/2012/09/11/basic-multicast-part-4-pim-sparse-mode-bsr-and-multicast-security/