Routing change question

Just wanted to think out loud on this and make sure Im not missing something before I push this out.

We have two connections to our MPLS WAN. connection 1 = DC1, connection 2 = DC2

Our core routers at each DC learn our external networks from the providers WAN routers at each DC. These routes are then redistributed into OSPF, and shared betweent he two DCs (theres 2 1gig fiber between them).

Now the problem is, that traffic from DC1 will always go out DC1's WAN router, since it never installs the 2nd route out of DC2 since BGP is the prefered route over OSPF (and is only peering with DC1s connection). Even though the OSPF database has both routes, BGP has a lower distance and therefore is installed.

So there are a couple of ways to solve this, but the quick and easy way I thought would be to just increase the distance on BGP, so that OSPF gets installed in the routing table, then I have both routes at my disposal to manipulate the way I want (the goal is to utilize DC2 more with some traffic from DC1)

Is there any reason this wouldn't work before I try it? Even though BGP wont be in the routing table, it will still redistrubte the learned routes into OSPF for it to use right?

I guess Im just paranoid of breaking something (internal or external haha).

Find more posts tagged with

Save $250 on 2025 certification boot camps from Infosec!

Book now with code EOY2025

Button

Comments

keenon

GT-Rob wrote: »

Ok, so Visio isn't on this computer, so its awesome MSPAINT to the rescue!

The 'core' of the network, is the same on each side. Most vlans are configured on both sides, but the active router is for most of them on the DC1 side (sorry, DC1 is on the left lets say), since thats where 95% of the traffic exists.

So, according to the diagram, 102 and 202 learn the remote site networks via the WAN routers, but only from the one they are connected to. These routes are redistributed into OSPF, so 102 and 202 eventually do learn both routes out, but because BGP has a lower distance, thats whats installed.

Now, since my DC1 router is basically maxed out some days while the DC2 router sits idle, im looking for a way to force traffic that hits 102, out the 202 route to WAN2.

One solution would be to simply make 202 (or 203) the HSRP active router for a few subnets. The problem with that is since most people and servers are in DC1, that will waste bandwidth between the sites.

Another solution is for 102 to peer with DC2's WAN router, but involving the ISP is the last thing I want to do.

Another solution is to policy route (which I am doing for a few things now), but we lose the dynamic failover that a routing protocol offers, and is not a great long term solution. Im also not sure what the performance impacts of this is.

The solution I was thinking of was simply giving OSPF a lower distance (or BGP with a higher), so that the OSPF routes make it into the RIB, and then I can edit the metrics to send certain vlans where I want them.

*edit* also, this is OUTBOUND traffic I am trying to manipulate. Inbound traffic is not a concern.

I would like to see a snippet of the configs from both the routers running bgp. mainly the ospf and bgp configuration parts. Are the wan1 and wan2 your routers or the ISP routers?

networker050184

ColbyNA wrote: »

Here's what I think would be best:

You run iBGP on the link between your two BGP routers, then you can do whatever you want with the traffic with weight or LP, basically what networker said in the beginning, lol. Then you don't need to mess with the BGP BD stuff and you have complete control through BGP.

Edit: With these being iBGP routes you might run into an issue with AD again lol.

You wouldn't need the BGP routes in OSPF at that point. You just need the internal reach ability with default to the BGP routers in OSPF. Then once the traffic gets to the edge routers they can route it out or across to the other DC depending on how you manipulate BGP.

ColbyG

networker050184 wrote: »

You wouldn't need the BGP routes in OSPF at that point. You just need the internal reach ability with default to the BGP routers in OSPF. Then once the traffic gets to the edge routers they can route it out or across to the other DC depending on how you manipulate BGP.

Yea, I was thinking you could just filter them, but your point definitely makes more sense. Just stop redistributing them and you have no issues. Boom, thread solved!

Teamwork, rofl.

GT-Rob

What edge routers? If I don't have the routes in OSPF, the how does 102 learn about WAN2? WAN1 and 2 are provider routers, set to just advertise in the external routes.

iBGP with a lower AD would work lol.

I don't have the exact configs handy sorry, and don't have my VPN client on this PC so I can't grab them. Nothing really fancy though, BGP is peering with the WAN router, OSPF is running on a handful of networks, all in area 0, with BGP and static routes being redistributed in.

ColbyG

I thought you were redistributing BGP into OSPF? That's why you wanted to raise the AD of BGP?

Forsaken_GA

GT-Rob wrote: »

One solution would be to simply make 202 (or 203) the HSRP active router for a few subnets. The problem with that is since most people and servers are in DC1, that will waste bandwidth between the sites.

If the goal is to keep your usage of the DC1 link down, I think this is probably your best option. Yeah, your intersite link will get a bit more usage than normal, but that's going to happen regardless of whatever traffic manipulation you do since the core problem seems to be not enough pipe at DC1. At least this way, you have some say in what goes across the link, but you still get failover in the event of a link failure.

And I know it's low tech, but have you considered maybe relocating some gear from DC1 to DC2 to help even out your usage?

I think your best solution overall would be to add links from WAN1 to 202 and WAN2 to 102, but I don't know if that would be feasible or cost effective for you

ColbyG

Really? I'm not a fan of having an L2 link there. I'd prever a point to point routed link and BGP for the traffic engineering.

networker050184

I'd go routed link too. You don't have to worry about broadcast or unicast flooding, no STP. None of that fun L2 stuff that likes to bite you in the ass out of no where. If there is a specific reason you need L2 connectivity for some crazy application or something then I'd do a L2 trunk with only that VLAN and your routed VLAN allowed. Then use SVIs to form OSPF and over the routed VLAN and route everything else. That will at least keep your STP and broadcast domains down to a minimum. Its not like you are gaining anything really with HSRP there. If the link or one of the routers fails you are out of luck with fail over anyway.

GT-Rob

The link is pretty idle between the centers, maybe 20% at most since the SANs have their own link for backups and such.

I mean theres no too much harm in Policy routing the big talkers over to DC2, which is what I am doing now, and just waiting until I get my own routers in there. We are also putting in 10g blades switches with 10g cards in the core switches, so who knows what the requirements will be tomorrow!

Either way, Ill lab this out and see what kinds of solutions I can come up with. Can't really "test" out much on our production network lol

EDIT btw Colby I am digging the blog, good stuff. When are you heading for the CCIE?

ColbyG

GT-Rob wrote: »

EDIT btw Colby I am digging the blog, good stuff. When are you heading for the CCIE?

Thanks! Interesting question about the IE. Earlier today I decided that I'm not close to ready for v4. I got the new INE Troubleshooting WBs, I felt like a lost puppy. Before jumping into show commands they would lay out the probably causes of X issue. If they had three potentials, I was lucky to think of even two of them. Made me realize that I need a good amount more experience before attempting that monster. V3 seemed hard enough, but adding troubleshooting changes everything for people like me with limited experience (as much as it sucks for me, it's great for the cert, so I can't complain too much).

I'm going to get serious about the DP and then maybe move onto the IP.

/shitnoonecaresabout

APA

Wow.. I love this stuff... been a while since a proper curly question popped up on here

My head hurts reading some of the replies but all good...

Here are my thoughts so far.....

- L2 trunk between your internal devices are fine.... Having a dedicated SVI used for routing between the two devices achieves what a dedicated PtP routed link would achieve... Plus the added bonus of carrying your trunked Vlans between the devices for your HSRP usage... etc

- My first thought was why don't you use the 'backdoor' trick with BGP and then I scrolled down and Colby had mentioned it (great minds think a like!).... However that would mean that your internal link would be used for routing traffic that would otherwise have been routed out of the WAN link... The RIB would no longer contain routes via the WAN Link that you have specifically noted as 'backdoor' eligible.... I get the feeling you want to load-balance between eBGP and the internal OSPF link right???

-Totally agree with Networkers statements about iBGP between your internal devices to allow them to learn routes between each other, however the AD of the eBGP routes will then become 200 when passed between the iBGP peers you will have to manipulate BGP to have a deterministic outcome that you are happy with... BGP Table\RIB wise.

I'm intrigued and hopefully will come back with an answer tonight..

APA

networker050184 wrote: »

Also, only routes from the routing table will be redistributed from BGP to OSPF as far as I know.

Correct.... redistribution is still a form of advertisement.... for BGP to advertise routes it must be in the RIB not just the BGP table.

GT-Rob

APA wrote: »

Correct.... redistribution is still a form of advertisement.... for BGP to advertise routes it must be in the RIB not just the BGP table.

This is what I was worried about, and what originally had me post.

As I said, I do have a budget to put in my own routers at each site, (along side some riverbeds) which is something I want to do for a number of reasons. This way I could stop BGP at those, advertise those routes into OSPF, and then let make the routing decision based on OSPF (obviously DC1 will still prefer WAN1 since it is closer, but I will just edit the routes as I see fit).

The other solution would be to make a BGP peering with WAN2 from DC1, so that it learns both routes via BGP. I guess this is the PROPER solution, however anyone who is a customer of Bell will understand

Anyway, Im glad you guys enjoy it lol. Ill be sure to bring more of it here!

ColbyG

APA wrote: »

- L2 trunk between your internal devices are fine.... Having a dedicated SVI used for routing between the two devices achieves what a dedicated PtP routed link would achieve... Plus the added bonus of carrying your trunked Vlans between the devices for your HSRP usage... etc

Passing VLANs across this link is definitely not the same as a routed p2p. I don't think carrying multiple VLANs across that link is a bonus, I think it's a bad thing. Unnecessary traffic is traversing that link. And HSRP here really doesn't do much, if that link is down so is HSRP.

GT-Rob

Yes but theres two links.

The redundancy design is that all distribution/access switches are connected to both cores at each site. So if SW_102 goes down, 103 picks up.

Actually now that I think of it, you are right. 202 will never work as a gw for hosts in DC1, since the only way it would become the GW is if both switches were down, putting any host offline anyway.

networker050184

GT-Rob wrote: »

Yes but theres two links.

The redundancy design is that all distribution/access switches are connected to both cores at each site. So if SW_102 goes down, 103 picks up.

Actually now that I think of it, you are right. 202 will never work as a gw for hosts in DC1, since the only way it would become the GW is if both switches were down, putting any host offline anyway.

That is what I was thinking. I don't think you really get much out of a L2 link here. It has more negatives than positives IMO. Routing is easier to control and A LOT easier to troubleshoot than L2 for me anyway.

I'm not sure why people are worried about the AD in this situation. If the external routes are ONLY in BGP (ie you have just a default to your BGP routers in OSPF) then AD will never come into play. You would only have those routes in a single routing protocol. BGP would also be very easy to manipulate what traffic to send where. You can also manipulate your inbound traffic to keep async routing down which is what would happen if you went with an outbound only solution.

What ever you come up with let us know!

APA

ColbyNA wrote: »

Passing VLANs across this link is definitely not the same as a routed p2p. I don't think carrying multiple VLANs across that link is a bonus, I think it's a bad thing. Unnecessary traffic is traversing that link. And HSRP here really doesn't do much, if that link is down so is HSRP.

What are you talking about? With the link being a Layer2 trunk and you using one of the VLANs passed across that trunk as a 'Virtual Point-to-Point' link acting as if a physical circuit would.... I don't understand how you can't get the same benefits as having a solely dedicated L3 Point-to-Point link? Still form an adjacency over it... still pass routing information over it...?

You just aren't locking the physical circuit to L3 operations...

Flooding, STP issues etc can all be prevented\mitigated if you provisiong the trunk link correctly with appropriate trunk configuration as per best practice trunk deployments. (Trunk Security, Change Native VLAN etc...)

If he wants to spread gateway load between his two cores its possible via HSRP but I do agree with the link down the point of HSRP becomes useless and carrying VLANs over the L2 trunk becomes a trivial operation. So in this situation the L2 trunk for HSRP becomes a nice to have not a need to have.

ColbyG

APA wrote: »

What are you talking about? With the link being a Layer2 trunk and you using one of the VLANs passed across that trunk as a 'Virtual Point-to-Point' link acting as if a physical circuit would....

Yea, you'll still be able to run everything across it, but my point is that you're also passing L2 information across it with whatever VLANs are on the trunk. I just don't see the need for an L2 link here.

APA

networker050184 wrote: »

I'm not sure why people are worried about the AD in this situation. If the external routes are ONLY in BGP (ie you have just a default to your BGP routers in OSPF) then AD will never come into play. You would only have those routes in a single routing protocol. BGP would also be very easy to manipulate what traffic to send where. You can also manipulate your inbound traffic to keep async routing down which is what would happen if you went with an outbound only solution.

What ever you come up with let us know!

I see where you're going with this.... that was my initial thought as well but for some reason I get the feeling Rob wants to pass those BGP routes between sites...

If he does then AD does come into play whether he redistributes OSPF into BGP or uses iBGP...

Rob is there anything preventing you from carrying out what Networker mentioned above via OSPF defaults to the bGP routers?

APA

ColbyNA wrote: »

Yea, you'll still be able to run everything across it, but my point is that you're also passing L2 information across it with whatever VLANs are on the trunk. I just don't see the need for an L2 link here.

Haha.... again we are on the same page but just needed to clarify..

See my post above about the L2 trunk being a nice to have not a need to have for Rob

APA

My turn to run to the gym now..... but thinkin out aloud really quickly...

I see you as having three options....

1) iBGP between yor two switches.... learn the networks that way.....manipulate AD so routes via iBGP would be preferred.

2) Default routes via OSPF...leave external networks in eBGP... Metrics to default-route next-hops would have to be identical to ensure both default routes are installed and load-balanced over.... otherwise static defaults would suffice to provide the identical metrics...

3) If you must redistribute BGP into OSPF as you are already doing possibly play around with the AD given to external routes sent into the OSPF domain by the ASBR.

That's just me thinking out aloud...

GYM TIME....

Forsaken_GA

APA wrote: »

My turn to run to the gym now..... but thinkin out aloud really quickly...

2) Default routes via OSPF...leave external networks in eBGP... Metrics to default-route next-hops would have to be identical to ensure both default routes are installed and load-balanced over.... otherwise static defaults would suffice to provide the identical metrics...

If his entire goal is to send some DC1 traffic out WAN2, then using a default route in OSPF to get egress traffic to the BGP routers and establishing a peering session between WAN2 and DC1 is his best option. That will let him get back to the norm of letting his IGP handle his internal routing, and influencing his egress traffic only via BGP. That way if he wants some traffic to go out WAN2, he can just send it directly, if the link goes down it'll go back to using WAN1, and you don't have to fiddle with the IGP to route egress traffic

APA

Forsaken_GA wrote: »

If his entire goal is to send some DC1 traffic out WAN2, then using a default route in OSPF to get egress traffic to the BGP routers and establishing a peering session between WAN2 and DC1 is his best option. That will let him get back to the norm of letting his IGP handle his internal routing, and influencing his egress traffic only via BGP. That way if he wants some traffic to go out WAN2, he can just send it directly, if the link goes down it'll go back to using WAN1, and you don't have to fiddle with the IGP to route egress traffic

Agree..... he would need to ensure that the peering link between WAN2 and DC1 has routes manipulated accordingly so that traffic wouldn't be sent out WAN1 but rather sent across the IGP and then out WAN2... (this is assuming a direct link between DC1 and WAN2 isn't provisioned or that the path\next-hops for the eBGP peering isn't already directed across the IGP)

Either way Rob... let us know what final solution you decide upon as really the ball is in your court

Its 42 degrees in Perth today... I'm going to kick back at the beach, take in the scenery and work on this lovely complexion I've got going on

GT-Rob

Right now (apart from hating APA as its -5 here right now), Im going to wait and see how soon I can buy those routers. Im going to see if I can spend 2010's budget before February :P

What I would really like to do, is put my own router between WAN1 and SW_102, and take BGP, tagging, and some other L3 things off of the core, and more them up. Then the core switches are only running OSPF to learn those external routes from the new routers, and the routing tables get cleaned up in the process (and summarize some of my routes since my provider can't figure that one out).

On Monday I will still lab this out though and post my findings!

APA

Right now (apart from hating APA as its -5 here right now)

Don't hate the playa.... hate on Momma Nature.... ;D

Quick Links

All Categories
Recent Posts
Activity
Unanswered
Groups
Best Of