Routing change question

GT-Rob

Just wanted to think out loud on this and make sure Im not missing something before I push this out.



We have two connections to our MPLS WAN. connection 1 = DC1, connection 2 = DC2

Our core routers at each DC learn our external networks from the providers WAN routers at each DC. These routes are then redistributed into OSPF, and shared betweent he two DCs (theres 2 1gig fiber between them).

Now the problem is, that traffic from DC1 will always go out DC1's WAN router, since it never installs the 2nd route out of DC2 since BGP is the prefered route over OSPF (and is only peering with DC1s connection). Even though the OSPF database has both routes, BGP has a lower distance and therefore is installed.



So there are a couple of ways to solve this, but the quick and easy way I thought would be to just increase the distance on BGP, so that OSPF gets installed in the routing table, then I have both routes at my disposal to manipulate the way I want (the goal is to utilize DC2 more with some traffic from DC1)

Is there any reason this wouldn't work before I try it? Even though BGP wont be in the routing table, it will still redistrubte the learned routes into OSPF for it to use right?

I guess Im just paranoid of breaking something (internal or external haha).

networker050184

Is there a reason you can't set up an iBGP peering between the two? I think that would be an easier way to do things rather than mess with AD.

Also, only routes from the routing table will be redistributed from BGP to OSPF as far as I know.

GT-Rob

networker050184 wrote: »

Is there a reason you can't set up an iBGP peering between the two? I think that would be an easier way to do things rather than mess with AD.

Also, only routes from the routing table will be redistributed from BGP to OSPF as far as I know.

That would be the 'better way', but it would mean involving the ISP which is such a pain. It would also be nice if I controlled the WAN router and could just run OSPF right off of it (and have BGP not come internal at all), but again, more pain than its worth.


The routes would still be learned by BGP, and be in the BGP table, just like the OSPF routes are in the database, but I want to install the OSPF routes instead of the BGP routes.

ColbyG

See if this helps:

BGP Backdoor Lab

GT-Rob

^ although that is an interesting trick I dont think I was aware of, its not quite what I am trying to solve. The internal routing is fine, as BGP is only being used to bring those external site's in, and OSPF is used between the two DCs for internal traffic.


The problem is traffic originating from DC1, goes out DC1, and traffic originating from DC2 only goes out DC2. The problem with this is that 95% of my outbound traffic comes from DC1, and DC2 goes unused. Right now I just use policy routing to force a couple of big talkers out DC2, but its hardly a dynamic solution.



In any case, Im going to hook a couple of spare 2800s up and see what the behavior is like, as it would be a good exercise either way. Ill post my findings after the weekend.



FWIW I didn't design this network haha. Just trying to fix it

ColbyG

Hmm. I don't think I'm fully understanding what you're trying to do. If you just want an OSPF route in the table instead of a BGP route, that will accomplish it. If you're trying to accomplish some load balancing/traffic engineering, yea, that's not what you want.

You want DC1 to kind of load balance across its WAN router and DC2?

GT-Rob

ColbyNA wrote: »

Hmm. I don't think I'm fully understanding what you're trying to do. If you just want an OSPF route in the table instead of a BGP route, that will accomplish it. If you're trying to accomplish some load balancing/traffic engineering, yea, that's not what you want.

You want DC1 to kind of load balance across its WAN router and DC2?

Not necessarily load balance, but at least give me some control over the routing a little more. Since BGP is only aware of the one route, I can't even manipulate it to use the other DC for some routes. The only reason I want the OSPF routes, as it knows of both DCs (since I dont want to peer with both routers on each DC via BGP). Once I have both routes 'available', then I can start manipulating them to balance them a little better (by giving some subnets a better cost via DC2 for example).

So the end goal is to have each DC at least KNOW about both routes out, then go from there. And to do that (again, instead of iBGP peering), is to use the OSPF routes.

ColbyG

So you can BGP Backdoor the networks you want to send to DC2. That will stick those OSPF networks in your RIB and send the traffic the way you want. If the link to DC2 dies you'll fall back to the BGP route. But yea, easiest way to do it would be iBGP between the two, IMO.

GT-Rob

ColbyNA wrote: »

So you can BGP Backdoor the networks you want to send to DC2. That will stick those OSPF networks in your RIB and send the traffic the way you want. If the link to DC2 dies you'll fall back to the BGP route. But yea, easiest way to do it would be iBGP between the two, IMO.

Oh ok I see what you are saying now. Problem with THAT, is that I don't have control of the advertising routers. I just learn them, then redistribute them into OSPF.


Yes your way would work though if I applied that to the WAN router, so that they were advertised to my core routers as 'backdoor routes' (sounds like a porno). That way I am still learning the routes, but using OSPF to make the decision as it has a lower distance.


blah, maybe ill just policy route my traffic and hope nothing ever goes down lol. I guess I could always point the routes to a secondary HSRP address and give me a little bit of failover.


God this sounds like a CCIE task. Do this, but not this way, and don't change this.

ColbyG

You don't have to apply it to the WAN router. On your DC1, you apply this:

network xx.xx.xx.xx mask xx.xx.xx.xx backdoor

It's not advertising them as backdoor routes, it is treating those routes as backdoors, so it raises the AD to 200 (IIRC) for them. So you end up with your OSPF routes having a lower AD and being installed into the RIB. It's all local to the router with the backdoor link.

For policy routing, just use SLA to track the next hop, if it goes down have it fail to the normal routes (DC1 --> WAN router).

And yea, this is some goofy ****, but I love having to think and dissect stuff.

networker050184

It does sound like something that would be in a lab. Honestly even though it sucks to get the provider involved I think that is your best bet at getting this thing going right.

ColbyG

networker050184 wrote: »

It does sound like something that would be in a lab. Honestly even though it sucks to get the provider involved I think that is your best bet at getting this thing going right.

No way! This is perfectly do-able.

ColbyG

Why is the traffic so asymmetrical? What's behind the DCs? Why don't each of them get ~50% of the traffic instead of routing DC1 to DC2 then out. Why not half to DC1 then out, and half to DC2 then out?

keenon

GT-Rob wrote: »

Just wanted to think out loud on this and make sure Im not missing something before I push this out.



We have two connections to our MPLS WAN. connection 1 = DC1, connection 2 = DC2

Our core routers at each DC learn our external networks from the providers WAN routers at each DC. These routes are then redistributed into OSPF, and shared betweent he two DCs (theres 2 1gig fiber between them).

Now the problem is, that traffic from DC1 will always go out DC1's WAN router, since it never installs the 2nd route out of DC2 since BGP is the prefered route over OSPF (and is only peering with DC1s connection). Even though the OSPF database has both routes, BGP has a lower distance and therefore is installed.



So there are a couple of ways to solve this, but the quick and easy way I thought would be to just increase the distance on BGP, so that OSPF gets installed in the routing table, then I have both routes at my disposal to manipulate the way I want (the goal is to utilize DC2 more with some traffic from DC1)

Is there any reason this wouldn't work before I try it? Even though BGP wont be in the routing table, it will still redistrubte the learned routes into OSPF for it to use right?

I guess Im just paranoid of breaking something (internal or external haha).

Ok, I have this same setup. DC1 is primary and DC2 is secondary. BGP into ospf. If your doing this easiest way is to set the metric on the primary DC to ex 100 and DC2 to 300. This should install the DC1 routes as primary. I have a detailed posting on my blog about it. However I'm doing this with 2 ISPs each we have 2 connections in each DC.

router ospf x
redist bgp xxxxx metric 100

http://theciscotech.blogspot.com/2008/11/dual-carriers-dual-connections-and-dual.html

ColbyG

The metric isn't coming into play here, it's all AD.

GT-Rob

Im about to leave but just quickly, the reason they are so asymmetrical is DC1 is the head office, where DC2 is a sort of failover DC. All the servers, email, etc, are all here in DC1, while only a few things are in DC2. The HSRP active router for all server VLANs is in DC1 since 80% of users are located here, and it beats traversing the trunk (its actually a pretty big company, so 2gigs is quite small).


The reason for the need of balance, is I only have 3mb out each DC to the MPLS, and DC1 is constantly pegged, while DC2 is basically unused.


Keenon: Wouldn't the router still pick the BGP route over OSPF since it has a lower distance?

Colby: I see what you are saying. I thought it was for advertising a distance to other routers. So that accomplishes what I had in mind where I just need OSPF to have a lower distance than BGP so it gets installed. You are right though, the most fun I have at work is trying to solve ridiculous problems like this lol


Anyway, I have already lugged up 4 2800s on to my desk so I will set this up either this weekend or monday, as there is a few things I want to test here. Im going to lab up the backdoor option and see if it accomplishes what I am looking for, and if so put it into production next week.

networker050184

Time for a diagram. Might help us get an idea exactly whats going on. I still don't see why you can't just set up an iBGP peer or peer both DCs to both edge routers.

Kaminsky

ColbyNA wrote: »

Why is the traffic so asymmetrical? What's behind the DCs? Why don't each of them get ~50% of the traffic instead of routing DC1 to DC2 then out. Why not half to DC1 then out, and half to DC2 then out?

Exactly. Big question here is, are your two DCs exact mirrors of each other ? That's kinda fundamental to what you are asking us to help you with and why you would want to route to one, the other or both.

BTW.. not having access to the carrier's routers is a biatch ain't it. Simple solution, stick your own routers as the next physical link from each of them and do what you want between your two routers from there on in. There's no reason you should ever need to touch a carrier's management A end router unless you hadn't slapped the guy who ordered the link hard enough beforehand ! < this is an very often overlooked but immensly important part of planning a network - let inept people do this ordering and you live with the grief of their ineptness for years to come - carrier/telco sales people are very, very wiley critters the world over >

ColbyG

Yea, a diagram would make this a lot easier. I was thinking about making one up and asking you if it's accurate, lol.

So you have a 2gb link between the two DCs?

keenon

heres a pic, is this about what you have?

ColbyG

Here's what I was picturing:



Forgive my terrible Visio skills and fruity colors.

One thing I'm not clear on though, if you have a trunk between the sites, where is OSPF coming into play? Especially if you're using HSRP on both sides, that makes me think you're passing your VLANs across the trunk, not routing it.

keenon

so if he is running ospf internally bgp is on the edge.

I would like to see a print of one of the routes in question from DC1 perspective and DC2. I would bet on something ospf related

ColbyG

Yes, BGP is on the edge, so his traffic from DC1 outbound is preferring the BGP routes (lower AD) and going straight to the ISP router. He wants some of his traffic to traverse the link to DC2 and go out that side.

networker050184

So are you routing between the two or is it just a L2 trunk? I'm confused.

keenon

ColbyNA wrote: »

Yes, BGP is on the edge, so his traffic from DC1 outbound is preferring the BGP routes (lower AD) and going straight to the ISP router. He wants some of his traffic to traverse the link to DC2 and go out that side.

odd in that case bgp drops routes that look to have originated from the same AS.. unless he not using the same AS number on both sides which would be the problem.

thus he should be using the same AS on both sides and updating his network advertising more specific on DC1 and summaries out of DC2 if he wants DC1 as primary. DC2 should use AS path prepending or MED to force the MPLS provider to prefer DC1

ColbyG

Dude you're all over the map. He's talking about OUTBOUND traffic, not incoming. Prepending/MED isn't applicable here (at least not for what he's asking).

GT-Rob

Colby that diagram is correct. The trunk between the two DCs is a L2 trunk between each other (its a 15km fiber between them). So logically, they are right next to each other. To add a bit of complexity, technically there are 2 6500 routers at each site, and the ones running BGP are ironically NOT the HSRP active routers.

OSPF is running to advertise other routes, including our very large DMZ (through another INET connection), various VPNs, voice routers, etc. Technically for this situation, it isn't needed at all since BGP gives us the external routes, and most internal networks are reachable via these core routers.


Networker: The only reason I dont want to make the additional peering right now is dealing with my ISP is like hitting my head against the wall. Ill be looking at a least a month to get something like that done.

Kaminsky: That is actually the plan lol. Ive got a budget to put in 2 routers at each site behind each of our ISP's router. That way ill be able to do run proper QoS, take some load off our core, these situations, etc.

Colby: We originally buried 6 fibers but they have slowly been used up (SAN, DMZ, etc), and we are down to just the 2 links left to connect the cores (technically one on each 6500 and then STP blks one, so really only 1gig).

ColbyG

Oh boy, my brain hurts more now, hahah.

Ok, first, why do you need to talk to your ISP if you want to run iBGP between your devices? This would simplify everything as everything would be BGP and you could manipulate it however you want.

Without doing that. These other 6500s, are they kind of the distribution switches? I don't see why you can't balance the traffic BEFORE it gets to the BGP routers (on your side) via routing protocols or even two static defaults.

If you could make a diagram with all four switches and kind of show the flow of traffic I think it would help us all grasp what exactly is going on.

Edit: Don't let this thread die, this is fun as hell.

GT-Rob

There are two switches at each side, again, just for redundancy (there probably see about 10% utilization each). iBGP between the two internal BGP routers would work, I was thinking more of making the DC1 switch a bgp peer with the DC2 WAN router and vice versa, since then it would just learn the routes from each and I could then manipulate what gets installed into the RIB. OSPF already knows of both routes, it just doesn't get a say.


Im about to head to the gym but will post a diagram tonight to show it more clearly.

GT-Rob

Ok, so Visio isn't on this computer, so its awesome MSPAINT to the rescue!




The 'core' of the network, is the same on each side. Most vlans are configured on both sides, but the active router is for most of them on the DC1 side (sorry, DC1 is on the left lets say), since thats where 95% of the traffic exists.


So, according to the diagram, 102 and 202 learn the remote site networks via the WAN routers, but only from the one they are connected to. These routes are redistributed into OSPF, so 102 and 202 eventually do learn both routes out, but because BGP has a lower distance, thats whats installed.


Now, since my DC1 router is basically maxed out some days while the DC2 router sits idle, im looking for a way to force traffic that hits 102, out the 202 route to WAN2.


One solution would be to simply make 202 (or 203) the HSRP active router for a few subnets. The problem with that is since most people and servers are in DC1, that will waste bandwidth between the sites.

Another solution is for 102 to peer with DC2's WAN router, but involving the ISP is the last thing I want to do.

Another solution is to policy route (which I am doing for a few things now), but we lose the dynamic failover that a routing protocol offers, and is not a great long term solution. Im also not sure what the performance impacts of this is.


The solution I was thinking of was simply giving OSPF a lower distance (or BGP with a higher), so that the OSPF routes make it into the RIB, and then I can edit the metrics to send certain vlans where I want them.


Anyway, I need to sneak in a run before the gym closes, so Im off now lol


*edit* also, this is OUTBOUND traffic I am trying to manipulate. Inbound traffic is not a concern.

ColbyG

Here's what I think would be best:



You run iBGP on the link between your two BGP routers, then you can do whatever you want with the traffic with weight or LP, basically what networker said in the beginning, lol. Then you don't need to mess with the BGP BD stuff and you have complete control through BGP.

Edit: With these being iBGP routes you might run into an issue with AD again lol.