Add redundancy to ISP facing router

mzinz

I attached a diagram.

Previously had 1 router connecting between service provider (Ethernet hand-off to MPLS cloud) and network core. We are now adding a second router for redundancy. Using EIGRP internally to choose a path to MPLS network.

What is typically done for the connection to the MPLS network? Right now I just have a single /30 IP address from the provider.

networker050184

Unless the provider is going to give you another handoff your second edge router isn't really going to provide much redundancy for you in this scenario. You'd probably be better off just keeping it as a spare.

shodown

for your setup you would run HSRP between the 2 routers, but like it was said above you have a much larger chance of failure on the ISP site than 2 internal routers. If they give you another hand off you can run IP SLA(option 1) and you won't have to get too fancy with the configs.

Option 2
You can get 2 hand off's and run BGP with 1 provider, I actually set this up a few days back. You will have your own private AS and run IBGP between your 2 devices and bring in a default route and set local pref to take the preferred path out.


Option 3
If you decide to go to another provider for redundancy you will have to obtain your own AS and public IP space to run BGP with the providers.


I may have made a mistake as I'm typing this out fast, but someone here will back me up in my thinking

mzinz

shodown wrote: »

for your setup you would run HSRP between the 2 routers, but like it was said above you have a much larger chance of failure on the ISP site than 2 internal routers. If they give you another hand off you can run IP SLA(option 1) and you won't have to get too fancy with the configs.

Option 2
You can get 2 hand off's and run BGP with 1 provider, I actually set this up a few days back. You will have your own private AS and run IBGP between your 2 devices and bring in a default route and set local pref to take the preferred path out.


Option 3
If you decide to go to another provider for redundancy you will have to obtain your own AS and public IP space to run BGP with the providers.


I may have made a mistake as I'm typing this out fast, but someone here will back me up in my thinking

Thanks for the replies.

As for option 1, are you proposing that I use HSRP on the internal side only, and IP SLA to failover if the provider hand-off goes down? Or HSRP on both internal and external?

mzinz

Also - why would it make more sense to run HSRP instead of letting a routing protocol (EIGRP) decide the path?

I had chosen EIGRP so that I could keep the connection between the switch and routers /30

shodown

Ok I see what you are doing. You are running EIGRP on your core switch. For some reason I thought you were doing something else(reading is fundamental showdown,lol). So yes you can just run EIGRP there. You might not even have to run it below. in the other setup I made. My next question for you is why are you guys using 2 routers.

jason_lunde

mzinz wrote: »

Also - why would it make more sense to run HSRP instead of letting a routing protocol (EIGRP) decide the path?

I had chosen EIGRP so that I could keep the connection between the switch and routers /30

Are you talking about best path to an outside network? or best path from the inside to the closest gateway? Those routers really have no metrics to hand eigrp for outside destinations since the are both connected to the same external provider (Assuming these are geographically in the same location). you are really looking at first hop redundancy from what I am looking at.

mzinz

shodown wrote:

Ok I see what you are doing. You are running EIGRP on your core switch. For some reason I thought you were doing something else(reading is fundamental showdown,lol). So yes you can just run EIGRP there. You might not even have to run it below. in the other setup I made. My next question for you is why are you guys using 2 routers.

Haha - no worries The two router question is a great point. It would have made more sense to use switches since it's a Ethernet hand-off, but the hardware was already purchased when I entered the contract. Also, I was keeping the diagram simple, it is handling some other jobs too, (VPN, QoS).

The purpose of having the second router is purely failover, in case the first one dies.

jason_lunde wrote: »

Are you talking about best path to an outside network? or best path from the inside to the closest gateway? Those routers really have no metrics to hand eigrp for outside destinations since the are both connected to the same external provider (Assuming these are geographically in the same location). you are really looking at first hop redundancy from what I am looking at.

Assuming I get a second hand-off from the provider, the paths to the MPLS cloud will be equal. The only reason I'm using EIGRP is in case one router goes down, we have a backup path via the second router. This may not be the best way to do this - so if not, then I'm all ears for suggestions. But it's not so much about choosing the best path (since they're the same), it's about having a backup route (feasible successor).

jason_lunde

Do you get any routes from your MPLS provider? What is your MPLS used for? i.e. inter-site connectivity, internet access, or both?

shodown

1 thing I'm going to add in my experience of doing this is that i have seen more circuits die than Redundant routers. I guess you already got the gear so might as well do something with it, unless you can get your partner to credit u for the gear and get something else you guys need. We do it all the time for customers.

networker050184

I think you will just be adding more complexity without much gain by adding a second router there. If you have one ISP hand off then the second router really only provides redundancy for one cable (the one from the core to the existing router). Not worth it IMO.

jason_lunde

networker050184 wrote: »

I think you will just be adding more complexity without much gain by adding a second router there. If you have one ISP hand off then the second router really only provides redundancy for one cable (the one from the core to the existing router). Not worth it IMO.

This is where i was headed with this conversation as well.

shodown

networker050184 wrote: »

I think you will just be adding more complexity without much gain by adding a second router there. If you have one ISP hand off then the second router really only provides redundancy for one cable (the one from the core to the existing router). Not worth it IMO.

Thanks for making it simple. I was trying to say this couldn't quite type it out.

mzinz

Thanks for the replies guys.

We don't get routes from the MPLS provider. We have about 200 remote branches connected, all within the same contiguous class B subnet. (So we currently have a static route for something like 192.168.0.0/16, with each site having a /24 internal LAN).

The reason they wanted a second router is because of the "what if it dies scenario". We would be completely down, as would all of our remote offices, for who knows how long. This way, even if something happens to the circuit, it isn't our fault.

I do understand what you're saying with the complexity and risk/reward issues. What would be a more suitable substitute? Maybe a couple stacked 3560s? It probably would be possible to get a credit for the gear.

I guess I thought that HSRP was for scenarios just like this? Would it just make more sense if we had a second provider, as opposed to two circuits from the same single provider?

EDIT: Or, alternatively, we could have the provider connect straight to our 3750 stack core, and rule out an intermediary device completely. I suppose this entire post is more of a fundamental design question than anything.

shodown

if you have that many sites that are relying on you I would get a connection from 2 providers(make sure you get path diversity) and run BGP and have your own Private AS inside. That way you can always be seen by the remote sites. I can get more into detail when I get home as I have a few fires I'm putting out now at work.

networker050184

mzinz wrote: »

The reason they wanted a second router is because of the "what if it dies scenario". We would be completely down, as would all of our remote offices, for who knows how long. This way, even if something happens to the circuit, it isn't our fault.

I'd just keep a hot spare which will do you just as much good. If the first router dies and it is the only one with a link to the ISP you are SOL anyway.

mzinz wrote: »

I do understand what you're saying with the complexity and risk/reward issues. What would be a more suitable substitute? Maybe a couple stacked 3560s? It probably would be possible to get a credit for the gear.

That could work out as long as you don't need any features of the router.

mzinz wrote: »

I guess I thought that HSRP was for scenarios just like this?

Not really. HSRP is suited to provide first hop redundancy for hosts. It allows you to not have to go changing the default gateway on devices when a router/switch dies. You really don't get anything from it when trying to do router to router redundancy. A routing protocol with fail over is what you would be looking for in this scenario as next hops are updated automatically anyway. It could help somewhat if you have static routes, but you want to avoid those if you can use a routing protocol instead.

mzinz wrote: »

Would it just make more sense if we had a second provider, as opposed to two circuits from the same single provider?

That would be my choice. Unless you specifically get a special design, another circuit will most likely ride the same equipment in your market at some point and fail the same time the other would if there was a cable cut or something.

mzinz wrote: »

EDIT: Or, alternatively, we could have the provider connect straight to our 3750 stack core, and rule out an intermediary device completely. I suppose this entire post is more of a fundamental design question than anything.

Again, if you don't need any router features (NAT, VPN etc.) this would work out pretty well for you. You mentioned VPN so thats probably not the best fit for you though. You could always hang the VPN router off the core and use it like a VPN on stick scenario if you really wanted to though.

mzinz

shodown wrote: »

if you have that many sites that are relying on you I would get a connection from 2 providers(make sure you get path diversity) and run BGP and have your own Private AS inside. That way you can always be seen by the remote sites. I can get more into detail when I get home as I have a few fires I'm putting out now at work.

Thanks for your help - more info would be great whenever you have a chance.

mzinz

networker050184 wrote: »

Again, if you don't need any router features (NAT, VPN etc.) this would work out pretty well for you. You mentioned VPN so thats probably not the best fit for you though. You could always hang the VPN router off the core and use it like a VPN on stick scenario if you really wanted to though.

The router does have other roles, but they aren't necesarily for the remote hosts. Doing something like VPN on a stick would work fine.

I would like to have some device between the 3750 stack and the provider hand-off for traffic policies, though. Maybe having two 3560's in a stack, running EIGRP in case of failover, would be the best option. I can apply ACL's on the L3 interfaces from the provider.

Functionally, it would be doing the same thing as the router, but I could probably get the 2 3560's for about the price of a single 3845.

Edit: If possible, I would get another circuit from another provider - I'm just not sure that is possible, budget wise. So I'm going for the "next best" option, really.

hasitha257

Originally Posted by mzinz
Would it just make more sense if we had a second provider, as opposed to two circuits from the same single provider?

Guys, aren't we talking about MPLS redundancy here? How is it possible to go with two ISPs for your MPLS redundancy?

Even if you had a redundant link to another ISP , how are you routing the traffic over to the remote sites?

I am just curious............appreciate any responses!!!

networker050184

hasitha257 wrote: »

Originally Posted by mzinz
Would it just make more sense if we had a second provider, as opposed to two circuits from the same single provider?

Guys, aren't we talking about MPLS redundancy here? How is it possible to go with two ISPs for your MPLS redundancy?

Even if you had a redundant link to another ISP , how are you routing the traffic over to the remote sites?

I am just curious............appreciate any responses!!!

Good catch. While you can get MPLS redundancy with two providers, it is a more complex scenario that involves cooperation between carriers. They would need to have an inter-AS VPN with each other.

ConstantlyLearning

mzinz wrote: »

Edit: If possible, I would get another circuit from another provider - I'm just not sure that is possible, budget wise. So I'm going for the "next best" option, really.

You should go with the solution that work's best. If that's getting a second provider, then plan that out and see if the budget allows for it.

mzinz

networker050184 wrote: »

Good catch. While you can get MPLS redundancy with two providers, it is a more complex scenario that involves cooperation between carriers. They would need to have an inter-AS VPN with each other.

Can you expand on this some, or provide a link if you have one? I don't have experience with that.

networker050184

There are quite a few flavors (options) for an Inter-AS MPLS VPN, but here is some info on option AB.

MPLS VPN - Inter-AS Option AB [Cisco IOS and NX-OS Software] - Cisco Systems

Its nothing you would have to worry about as the customer. The second provider you get a circuit from would need to already have an arrangement with your primary provider for this to be set up though.

We have a few set up with other providers, but honestly I'm not quite sure how popular these are so I have no clue what the chances are that your provider has something like this in place.

Another option is a backup IPSEC tunnel or DMVPN over the internet.

mzinz

networker050184 wrote: »

There are quite a few flavors (options) for an Inter-AS MPLS VPN, but here is some info on option AB.

MPLS VPN - Inter-AS Option AB [Cisco IOS and NX-OS Software] - Cisco Systems

Its nothing you would have to worry about as the customer. The second provider you get a circuit from would need to already have an arrangement with your primary provider for this to be set up though.

We have a few set up with other providers, but honestly I'm not quite sure how popular these are so I have no clue what the chances are that your provider has something like this in place.

Another option is a backup IPSEC tunnel or DMVPN over the internet.

Interesting. Thanks for the information. I think I'll call my provider and ask if it's something they would support.

For the second option, are you suggesting that the second router has a connection to the internet, and we have DMVPN set up on all remote branches, and that it uses the VPN link only in cases of failover? (ie: Use VPN if MPLS unaccessible)

networker050184

mzinz wrote: »

For the second option, are you suggesting that the second router has a connection to the internet, and we have DMVPN set up on all remote branches, and that it uses the VPN link only in cases of failover? (ie: Use VPN if MPLS unaccessible)

Yes, exactly. Depending on your network needs, something as simple as business DSL could provide a backup.

mzinz

networker050184 wrote: »

Yes, exactly. Depending on your network needs, something as simple as business DSL could provide a backup.

This would be an excellent solution... unfortunately we don't have any internet connections at the remote branches - just the MPLS.

networker050184

Your provider should be able to get you internet access over those same MPLS circuits without much trouble. I'm sure you'd have to pay some sort of MACD fee, but it could be minimal.

DSL isn't overly expensive to get into your branches either. If they want redundancy its going to cost something besides just one extra router to sit there doing nothing

mzinz

networker050184 wrote: »

Your provider should be able to get you internet access over those same MPLS circuits without much trouble. I'm sure you'd have to pay some sort of MACD fee, but it could be minimal.

DSL isn't overly expensive to get into your branches either. If they want redundancy its going to cost something besides just one extra router to sit there doing nothing

Fair enough

It seems like having a connection into the same MPLS cloud from a different provider would really be the best move, if possible. We don't need absolute redundancy at each and every site. Having a secondary router in case the main circuit goes down would be great though.

sides14

The greatest thing to remember about redundancy is that it needs to be truly isolated from the original path. Many providers will sell you a redundant link, but when you drill down to the detail (or have an outage), you find out pretty quick that the path is not redundant.

Story about redundancy. Had to "redundant" GigE circuits from a provider. Wild fire caused both links to go down. After further investigation, the circuits were on two distinct sets of fibers, but the first and second sets were OPGW on the same 250KV line - not very redundant or protected.