Multi-homing and Circuit Bonding

hermeszdata · April 2011

I am reviewing some information (Specifications) for a customer and I need a bit more in-depth understanding of the what's and how's.

The spec calls for a combination of T1 and DSL or Cable HSIA and that the connections be bonded. The idea is to have redundant connections so if one goes down the remaining takes the full load. Bandwidth Management (load balancing) is also a part of the equation. Because of the circumstances, they will be using the equivalent of /27 prefix-length.

So, from a Cisco perspective, how do we implement this? (Haven't gotten to that part of the ROUTE Study yet!) There is a debate (client internal) about whether there is a requirement of have different providers T1 and DSL (current estimates to bring Cable HSIA in excess of $30K due to extensive road bore and building penetration makes this option unrealistic) and the specs are not clear on that issue. I know from experience, that there is a single TELCO provider in any Metro area and the other providers in the Metro Area are actually re-sellers of that service, buying it at discounted rates and handling the Subscriber Support in place of the Actual provider.

Now, that was a mouthful to get out!

Back to the question(s). How do we accomplish this from a Cisco perspective? I know there are 3rd party devices that will do this, but I want to understand from the context of this forum and how we would implement a Cisco Solution.

They actually have 2xT1s (bonded) and will add a DSL subscription. They will have to expand their existing blocks to at least 25 statics. IF I understand correctly, should all services come from the same ISP, that ISP could/would take care of the Circuit Bonding with all circuits pointing to the same default gateway. Is this a somewhat accurate assessment or did I fall down and bump my head again?

In this particular case, the ISP has provided a router and the site connects to the ISP Device via FaEth to their Netgate (it's not clear whether the ISP would bring in a standalone DSL modem or provide a DSL module for its router.) So for the sake of argument, let's say I am providing the router with a WIC-2DSU-T1 and a WIC DSL module to accommodate the dual T1s coming from the SmartJacks. and the 2-wire DSL line. What would be the basic processes to implement the circuit bonding, multi-homing, and bandwidth management to achieve the desired goal(s)?

Explain it to me "Like I's A 6-year old!" not really, but you get the idea.

Thanks in advance. I have to discuss this on a conference call first thing in the morning.

Forsaken_GA · April 2011

To start out with -

I have absolutely no clue how to do this, I'd have to go research it.My method of load balancing has always been to acquire BGP capable circuits from providers and handle it through that. I'm guessing this is a situation where that's not feasible, however

With that being said... you're probably not going to find an answer on this forum, at least not before your conference call. You're much better off flexing your Google-fu, because this is the kind of question that when folks ask me about in real life, I explain there's a need to discuss my hourly rate before I can help them

hermeszdata · April 2011

Forsaken_GA wrote: »

To start out with -

I have absolutely no clue how to do this, I'd have to go research it.My method of load balancing has always been to acquire BGP capable circuits from providers and handle it through that. I'm guessing this is a situation where that's not feasible, however

With that being said... you're probably not going to find an answer on this forum, at least not before your conference call. You're much better off flexing your Google-fu, because this is the kind of question that when folks ask me about in real life, I explain there's a need to discuss my hourly rate before I can help them

I was able to find a couple fo things via Google. I'll be speaking with the ISP before the call and query about BGP. Implementation is well beyond the understanding ability of the conference call principles so this post is mainly for my education/edification.

This may all be a moot point because I will have to tell the Corporate VP that in order for the local site to meet the compliance requirements it will require a complete rework of their front-end network due to 5+ years neglect. The specs require "Enterprise Class Equipment" and they have a mixture of consumer, small business and enterprise equipment! Additionally, they have an unknown number of de-commissioned/still powered equipment, un-powered/still connected, cables gowing who knows where and who knows what else and will require demolition and re-cabling of their network closet as just a beginning to meeting compliance. What makes things worse is a site GM how wants everyone to believe that they are 90%+ compliant! IF I had a brain, I would probably walk away!

I start on the network upgrades for their sister site tomorrow afternoon installing new (they do not currently have) Router/firewall and Switch for their front end and a new switch and AP for their wireless network (12-15 APs) and when that is done we negotiate an Annual Support Contract.

Just another Day in the life" of me!

chrisone · April 2011

hermeszdata wrote: »

I am reviewing some information (Specifications) for a customer and I need a bit more in-depth understanding of the what's and how's.

The spec calls for a combination of T1 and DSL or Cable HSIA and that the connections be bonded. The idea is to have redundant connections so if one goes down the remaining takes the full load. Bandwidth Management (load balancing) is also a part of the equation. Because of the circumstances, they will be using the equivalent of /27 prefix-length.

So, from a Cisco perspective, how do we implement this? (Haven't gotten to that part of the ROUTE Study yet!) There is a debate (client internal) about whether there is a requirement of have different providers T1 and DSL (current estimates to bring Cable HSIA in excess of $30K due to extensive road bore and building penetration makes this option unrealistic) and the specs are not clear on that issue. I know from experience, that there is a single TELCO provider in any Metro area and the other providers in the Metro Area are actually re-sellers of that service, buying it at discounted rates and handling the Subscriber Support in place of the Actual provider.

Now, that was a mouthful to get out!

Back to the question(s). How do we accomplish this from a Cisco perspective? I know there are 3rd party devices that will do this, but I want to understand from the context of this forum and how we would implement a Cisco Solution.

They actually have 2xT1s (bonded) and will add a DSL subscription. They will have to expand their existing blocks to at least 25 statics. IF I understand correctly, should all services come from the same ISP, that ISP could/would take care of the Circuit Bonding with all circuits pointing to the same default gateway. Is this a somewhat accurate assessment or did I fall down and bump my head again?

In this particular case, the ISP has provided a router and the site connects to the ISP Device via FaEth to their Netgate (it's not clear whether the ISP would bring in a standalone DSL modem or provide a DSL module for its router.) So for the sake of argument, let's say I am providing the router with a WIC-2DSU-T1 and a WIC DSL module to accommodate the dual T1s coming from the SmartJacks. and the 2-wire DSL line. What would be the basic processes to implement the circuit bonding, multi-homing, and bandwidth management to achieve the desired goal(s)?

Explain it to me "Like I's A 6-year old!" not really, but you get the idea.

Thanks in advance. I have to discuss this on a conference call first thing in the morning.

In simplest terms you need to implement tracked routes. Practice with tracked routes, research it first and learn how to implement it. With track routes you can have one circuit your primary and secondary with a higher AD waiting in the background. Once the primary circuit goes down the secondary route will automatically take over. Once you have this set you will have 2 default routes, they will have different AD's. Your default routes will obviously point to the various circuits you have as next hop.

You can perform load balancing with route maps or QOS. Although i havent played much with load balacing with route maps or QOS the posibility is there. With route maps you would obviously divide your LANs and send them to different next hops, thus dividing the load on different circuits. However I am skeptical if this will work with the track routes or if this will nullify the tracked routes. Anyways i hope this gets you started on the correct path, i just brained storm off the top of the brain.

sides14 · April 2011

Couple of questions. Is the DSL circuit to be a back-up for the two bonded DS1s? Honestly, if you are looking for redundancy, I would go with the cable option. The T1s and the DSL would use the same B-Box which would definitely not provide any redundancy because the weakest link would be the box out on the street. Maybe consider one of the wireless T1 or broadband solutions to provide redundant service.

hermeszdata · April 2011

chrisone wrote: »

In simplest terms you need to implement tracked routes. Practice with tracked routes, research it first and learn how to implement it. With track routes you can have one circuit your primary and secondary with a higher AD waiting in the background. Once the primary circuit goes down the secondary route will automatically take over. Once you have this set you will have 2 default routes, they will have different AD's. Your default routes will obviously point to the various circuits you have as next hop.

You can perform load balancing with route maps or QOS. Although i havent played much with load balacing with route maps or QOS the posibility is there. With route maps you would obviously divide your LANs and send them to different next hops, thus dividing the load on different circuits. However I am skeptical if this will work with the track routes or if this will nullify the tracked routes. Anyways i hope this gets you started on the correct path, i just brained storm off the top of the brain.

Thanks Chris.

I'll look into this possibility as well. For the time being, after my pre-conference ISP discussion, I will most likely leave this to the ISP to implement (if possible) or to the company that manages the site's netgate 8100 (which if I remember correctly can provide load balancing - I've done so many Netgate, Radware, and Sonicwall installs - rack/stack/X-connect - I can't remember who did what with what).

I know, if tomorrows conference is successful, I will need to implement a simular solution at the sister site.

The ISP, same for both sites, have provided an Adtran router (don't remember the model) that also connects to their PBX. IF this had been a Cisco device I could ask that they do an EIGRP implementation and then set up unequal cost load balancing.

Gotta love the learn on the fly routine!

hermeszdata · April 2011

sides14 wrote: »

Couple of questions. Is the DSL circuit to be a back-up for the two bonded DS1s? Honestly, if you are looking for redundancy, I would go with the cable option. The T1s and the DSL would use the same B-Box which would definitely not provide any redundancy because the weakest link would be the box out on the street. Maybe consider one of the wireless T1 or broadband solutions to provide redundant service.

The requirement is load balancing over all subscriptions. This is not a backup or failover config. rather as originally described a redundant connection providing multi-homing, bandwidth control, and an assurance that if one line goes down they will still have connectivity.

The specs require 1xT1 plus a DSL/Cable subscription. With the construction costs involved, Cable is out of the question (as mentioned in my op, $30K min). I'm sure I will need to get on the phone with the controlling entity to get their take given the circumstances (review the information regarding TELCO and Re-Sellers).

As a former HughesNet Dealer/Installer I could not recommend Internet over satellite due to weather related problems. The same thing holds true with 3G/4G connections. Satellite and *G connections are OK if there are no other options or as an emergency failover, but definitely not a viable choice given the stated specs. If this site had cable in a previous life, the problems would be reduced. Unfortunately, this is not the case. Cable is also requiring multiple subscription categories that are not contractually possible at this time. termination of existing related contracts would be too costly.

EDIT: If the cost of compliance is too great, I would not be surprised to see the owners decide to drop the franchise and go to a different franchise brand!

tomaifauchai · April 2011

You could use PPP multilink and add any kinda circuits to the bundle. It does load balancing by default and if 1 circuit went down, the bundle still work. On the ISP side, i don't know how they could bundle the 2 circuits but hmm check those links

Cisco: Routers - Multilink T1 with diff ISP ?

Use Multilink PPP to combine multiple circuits into a single circuit with a single router interface | TechRepublic

Load balancing over BGP could also be possible by enabling the "maximum-paths" to 2. You could run private-as number if you can't afford a real as but the load balancing would be innefficient.

hermeszdata · April 2011

tomaifauchai wrote: »

You could use PPP multilink and add any kinda circuits to the bundle. It does load balancing by default and if 1 circuit went down, the bundle still work. On the ISP side, i don't know how they could bundle the 2 circuits but hmm check those links

Cisco: Routers - Multilink T1 with diff ISP ?

Use Multilink PPP to combine multiple circuits into a single circuit with a single router interface | TechRepublic

Load balancing over BGP could also be possible by enabling the "maximum-paths" to 2. You could run private-as number if you can't afford a real as but the load balancing would be innefficient.

This is not a Point-to-Point topology where a branch is connecting to HQ. This is a Hospitality situation where the site is providing guests with the ability to connect to the internet as well as the necessary connection for the franchise to do business. I really cannot offer more about the specifics as i am already pushing the edge of confidentiallity. Basically, the spec is intended to ensure two things:

Site network reliability
Guest User Satisfaction

but isn't this what out job is about! 100% uptime as is reasonably possible!

I have added the various links provided from all respondents to my favorites (properly cateloged) and saved any that were PDF links.

matthewmalk248 · July 2013

I'm actually doing this same job for a site in my area, same story with the gm wanting to be close to compliance while currently being no where near. Was thinking about just using a hardware firewall/UTM with dual WAN and built in load balancing and failover. I'm starting to feel I under quoted. How did you end up doing with this? Should I run as fast as I can?

xXErebuS · July 2013

What is the average type of traffic? Are you an e-commerce company with a TON of outbound traffic; or are you a normal company with a TON of inbound traffic (i.e. bunch of users using the internetz, downloading, etc).

The reason I ask is because all of the methods listed above for load balancing (multipath / different gateways) assume your traffic is mostly OUTBOUND.

Now from what you've said the site purposes, and what I've found to be out of most networks is that if you monitor your WAN interfaces you will see most of the bandwidth is INBOUND (makes sense right? your sending simple requests outbound and your content is delivered inbound).

You can accomplish "load balancing" via specific prefixes in BGP, otherwise you are at the mercy of the ISP (i.e. multihoming single provider, BGP attributes, etc), in which I WOULD NOT TRUST - we once had a provider appending AS #s so their link wouldn't be utilized unless necessary. I would recommend familiarizing yourself with BGP looking glasses and ask for full BGP from the ISPs.

Oh and depending on SLA; I would go with two seperate ISPs regardless of medium - what happens when there is a maintenance interval with ISP in your area?

Jackace · July 2013

I can't vouch for this particular product, but from what I have seen you want a device like the one in the link below.

https://www.mushroomnetworks.com/product/truffle?search=1

boobobobobob · July 2013

Get a firewall and have that do the load balancing for you, this is how most enterprise networks load balance between ISPs. If you have the budget run HSRP or GLBP for redundant cores but all IDFs will also have to be dual homed to the cores.

xXErebuS · July 2013

boobobobobob wrote: »

Get a firewall and have that do the load balancing for you, this is how most enterprise networks load balance between ISPs. If you have the budget run HSRP or GLBP for redundant cores but all IDFs will also have to be dual homed to the cores.

Not necessarily, you could have a L2 device between the FHRP addresses and be single homed to it.

I would say most enterprises use BGP and rely on that for ISP load balancing; but once again you can only do so much and if your return traffic is what is consuming your link; what you do between the ISP link and your end user doesn't matter much.

Kind of reminds me of what I've seen networker050184 post about several times - routing goes two ways....

boobobobobob · July 2013

xXErebuS wrote: »

Not necessarily, you could have a L2 device between the FHRP addresses and be single homed to it.

I would say most enterprises use BGP and rely on that for ISP load balancing; but once again you can only do so much and if your return traffic is what is consuming your link; what you do between the ISP link and your end user doesn't matter much.

Kind of reminds me of what I've seen networker050184 post about several times - routing goes two ways....

Doesn't that make the network not redundant?

it_consultant · July 2013

If you are pulling in diverse links (say DSL, cable, T1) and you want to dynamically load balance them, I have never seen this done without a load balancing device - either built into the firewall or on a separate linux based appliance. The watchguard pro software lets you load balance a couple of different ways, for example, I could have 1/3 of the connections go through one link (like a weighted round robin) and the other 2/3 go out the other link and then failover to one in case of failover. In that case, I could even track a public IP from each interface which would indicate a failure; which is important because the link may still be up but their might but their might be a routing issue in the POP causing issues, if it misses three pings on that interface it fails it out. Similar things can be done with ServerIrons from Brocade and I would think F5 could do it as well.

xXErebuS · July 2013

boobobobobob wrote: »

Doesn't that make the network not redundant?

Makes the access layer non redundant; you still have redundant cores. Is it optimal? No. Will it better fit a budget? Yes.

Multi-homing and Circuit Bonding

Comments