BGP confederations in production usage

tomtom1

Hi guys,

Revisited the use of BGP confederations tonight. Whilst it is a technique that works well and can be used alongside route reflectors to compliment it, it also "sort of" provides the same functionality, overcome BGP's rule of split horizon: A prefix learned via an iBGP neighbor cannot be advertised to another iBGP neighbor.

With the confederation you basically chop up your AS into smaller sub-ASes, invisible for external peers, but useable within the AS to also overcome the split horizon rule since for routers in different confederation sub ASes, eBGP rules apply. However, I'm having trouble coming up with a scenario where this can be used in a production environment. Where would this be a better choice than a RR or be used alongside a RR?

Hoping for your insights,

fredrikjj

tomtom1 wrote: »

Where would this be a better choice than a RR or be used alongside a RR?

If your AS becomes so big that your IGP can't handle it, you could theoretically separate the AS with confederation. That's pretty much it when it comes to things that you wouldn't be able to do with RR as well. I've only read books on this, but I get the impression that RR was widely adopted right away and became the de facto standard because it wasn't as disruptive to deploy as confederation. Additionally it only required that the RRs themselves supported the feature which must have made migration to it much easier compared to confederation which requires that every single router supports it. You would have to reconfigure every single BGP router in your network as well.

Personally, I don't really like RR because there seem to be way too many caveats with the design, and it just seems too hard to be certain that you aren't screwing things up. I don't feel like it's deterministic enough, but if I really tried to study it hardcore, maybe I would get it and get over my anxiety about it*. Confederation is much cleaner and its behaviour follows directly from normal BGP behaviour, more or less. Real life is real life though and there are probably tons of operational issues I'm not even aware of.

*Go read BGP Design and Implementation's route reflector design chapter if you want your confidence crushed.

tomtom1

Yes, that was my impression as wel, seeing as by using a confederation you need to reconfigure the router bgp process to reflect the AS of the subAS used within the confederation. I'm curious to hear other opinions.

networker050184

In my opinion confederations should never be used in production! We just got through cleaning up a network with confederations and boy was it ugly and really gained nothing from using them. Route Reflection is the way to go in my opinion. Not sure what all caveats fredrickjj is talking about, but as long as you have a nice hierarchy set up it's pretty simple. Most service provider networks I've seen is a pretty standard core reflects to distribution, distribution reflects to access. Some other setups are "gateway" routers reflecting internet to core routers and core to access. Most service providers these days do their VPN (VPNv4,v6,multicast families etc.) reflection with dedicated VPN reflectors that every VPN enabled router in the network peers to.

lrb

Never seen anyone use a confederation properly in a real network. Route reflectors are much more common in any of the networks I've worked in. Confederations read like a cool idea but IP and VPN RRs are much more common.

Dieg0M

Never seen confeds in production either. I have seen RR but we now try and keep away from them.

networker050184

Can't keep away from reflection in larger networks. Having thousands of iBGP sessions is pretty unrealistic.

powmia

Confederations come with significant caveats. The big one is that the CONFED_AS path is not taken into consideration for best-path selection, a hierarchical design is required with confeds in order to ensure deterministic forwarding paths... the same is true for RRs, but relying on IGPs for forwarding is expected in a single AS with RRs... not in something broken into multiple ASs. There's also less flexibility and potentially more routers with irrelevant state in your network, as most people are running MP-BGP, and your ASBRs would need to hold that info, not just your RRs and PEs.

Book answer: You would use a confed design in the same way you would use any other multiple AS design; when you are breaking up administrative domains (maybe just multiple IGPs, typically something more organizational). If that's the case, though... it is better to just get multiple public ASs if you organization is large enough. Picture this, you are Tier I provider with a sub-AS in North America, and another in South America. You peer with another provider in Europe from both locations. As far as their AS_path is concerned to reach one of your customers in South America, you appear to be one AS... so North American paths may be preferred. You wouldn't have this issue with public ASs.

My answer: If you just need to eliminate the full-mesh requirement, use RRs. If you need to create administrative boundaries, use multiple ASs. If you can't afford multiple ASs, use confeds.

powmia

As far as caveats for RRs... I assume that refers to cluster-id and originator-id They are pretty straight forward concepts, and though there is debate on how to set clusters up, the answer is straight forward when you look at it from a distance.

fredrikjj

I'm mainly talking about how it's possible to create forwarding loops if you don't know what you are doing, and the heuristic you are given to solve the problem is "follow the physical topology", but how do you guarantee that some failure scenario won't create a corner case that you haven't thought of. There has been a number of issues with RR over the years as well, like the oscillation MED problem, and now some IGP thing (When the Cure is Worse than the Disease: the Impact of Graceful IGP Operations on BGP | INL: IP Networking Lab), and probably a lot of other things that I'm not aware of. To an outsider it doesn't inspire confidence and the feature clearly wasn't well thought out or tested at the time. However, it's obviously proven and mostly works in real life. Don't interpret this me trying to be some know it all, because I'm not, and I've never been more confused about a feature that other people are fine with. I need to study it more.

networker050184

Well, there are pitfalls to any protocol or feature with bad design. Route Reflection is widely used today without issue.

reaper81

Normally RRs would be setup to be out of the forwarding path. It should be a control plane feature and these devices should not be involved in forwarding.

With virtualization being hot I would seriously consider running something as CSR1000v or XRv to run virtual RRs. That way if a RR goes down you could move it to another server.

When you get really big you might need to have several levels of RRs, which can get complicated.

Newer features overcome a lot of the disadvantages of RRs, features such as add-paths and best-external so that you can have multiple paths advertised via the RR.

I have never seen confeds in real life and would never use them. Like powmia said, if you want to divide your domains then use real AS numbers.

networker050184

reaper81 wrote: »

Normally RRs would be setup to be out of the forwarding path. It should be a control plane feature and these devices should not be involved in forwarding.

I've heard many arguments for and against this over the years and have worked on networks with both designs. Some vendors recommending having them in the path, some out. The vast majority of networks I have seen use inline though for internet routes.

Personally I like to keep my IP groups RR inline (VPN peering gets dedicated RRs with minimal update groups). Just a personal preference really as it works both ways obviously. I looked into the Quagga/Zebra RR stuff a while back but the code just wasn't mature enough at the time to run a service provider network on. Maybe something to look back into. Can run on cheap hardware.

tomtom1

networker050184 wrote: »

I've heard many arguments for and against this over the years and have worked on networks with both designs. Some vendors recommending having them in the path, some out. The vast majority of networks I have seen use inline though for internet routes.

Personally I like to keep my IP groups RR inline (VPN peering gets dedicated RRs with minimal update groups). Just a personal preference really as it works both ways obviously. I looked into the Quagga/Zebra RR stuff a while back but the code just wasn't mature enough at the time to run a service provider network on. Maybe something to look back into. Can run on cheap hardware.

Wow, guys, a lot detail here. Let me see if I can still understand it. By having them out of the forwarding path, you mean not using them as next hops, but rather just letting them perform the RR functionality on the control plane? As to the functionality of confederations itself, you guys pretty much confirmed what I was thinking. Thanks for that. It's great and humbling to see there is still very much to learn.

networker050184

Yep, you would basically have a dedicated router (better have at least two for redundancy!) sitting off to the side that the rest of your routers peer to. This dedicated router reflects the routes back down to the other routers. One of the things I don't really like about this model is filtering for aggregation. For instance you want section A routes in section A only and section B in B etc. For this you need to either create a bunch of different update groups (which means more processing power) or filter per neighbor. Just makes more sense to have a pair of inline routers doing the refection with a single update group and a few peers in my opinion. Always give and take though.