wan study for a small business

Bl8ckr0uter

Well guys I have a problem. Come the first of the year we are going to start looking into upgrading our WAN link. What we have now is severely lacking and with the plans we have (offsite replication of sql servers) we have to get a bigger link. My boss is thinking of either mpls or metro e. My counterpart is thinking mpls because it 'just works'. The cost for an mpls solution seems a bit outrageous especially for a company with only 70 users at the main site and 3 at a remote site.

The problem is I don't think I know enough about either technology to really know the pros and cons of both and thevsituations when you would want to use either one. Anyone have any good reading sources on them both? Also anyone have a really good comparison of the two technologies? I did Google and I found some good info but most of it seemed like marketing speak. Also for those of you who have been in this situation what are the types of questions you have asked to vendors? Any caveats or pitfalls?

Forsaken_GA

Well, it depends on the offering, and you'd need to look at what the companies themselves are offering. Some MetroE's are delivered over MPLS. MetroE is normally delivered as an end to end layer 2 circuit

MPLS can be delivered anyway the carrier is willing to do it. You could deliver a layer2 link via MPLS, or you could get a layer 3 link.

You need to identify specifically what it is you need before anyone can give you any real advice. Do you need a layer 2 link between the two sites? MetroE or VPLS will work fine. Are you going to have a router on each site and you need them to share routes? Ok, then you want an MPLS L3VPN

Personally, I prefer layer 3 links. Layer 2 links can do some interesting things... broadcast storms across WAN links are not fun

If you do not have a good solid grasp of layer 2 network design, I strongly suggest staying away from a layer 2 circuit.

If cost is an issue, consider just getting a normal bigger pipe on both ends, and then setting up an IPSEC tunnel between the two end points.

My company does SQL replication to multiple sites over the WAN, and we use layer3 MPLS vpn's. Unfortunately, our provider is somewhat unreliable, and MPLS circuits that are shared by the entire company for intersite traffic (not just replication) can get saturated *real* easy. So we have public internet circuits that connect to each other over IPSEC tunnels as our backup solution for when the MPLS circuit is having issues (you're going to want to bring up the issue of redundancy to your boss.... with database replication, a circuit going down for too long can make it nearly impossible for the remote servers to catch up and get back in sync. Seeing a database slave that is 10 million seconds behind the master is *not* a good thing)

Bl8ckr0uter

Forsaken_GA wrote: »

You need to identify specifically what it is you need before anyone can give you any real advice. Do you need a layer 2 link between the two sites? MetroE or VPLS will work fine. Are you going to have a router on each site and you need them to share routes? Ok, then you want an MPLS L3VPN

Personally, I prefer layer 3 links. Layer 2 links can do some interesting things... broadcast storms across WAN links are not fun


If you do not have a good solid grasp of layer 2 network design, I strongly suggest staying away from a layer 2 circuit.

We are a small shop with two sites. Right now, the two sites are connected with an ipsec vpn and that is fine. There is a bandwidth issue (since all of our voip traffic also goes across that same link, at least in relation to our second site) but there are so few people there it doesn't make a difference

What my boss wants to do is have replication to a third site (which doesn't exist yet) for backup purposes for our main database application. I am not sure if we need layer 2 or 3 connectivity. I have read that layer 3 designed networks has been the way to go because layer 3 protocols have better "protections" and self healing. <noob alert> What would be the advantage of going layer 3 vs layer 2 in your opinion (or vise versa)?

I am sure we would want a different internet link (with a different provider) for DR purposes as well as load balancing. To me we could easily solve this by getting a bigger primary internet pipe and then a seperate pipe for replication traffic but that could be because of my lack of knowledge of WAN solutions.

</noob>

Forsaken_GA wrote: »

If cost is an issue, consider just getting a normal bigger pipe on both ends, and then setting up an IPSEC tunnel between the two end points.

Exactly.

Forsaken_GA

Bl8ckr0uter wrote: »

What would be the advantage of going layer 3 vs layer 2 in your opinion (or vise versa)?

Generally speaking, I find layer 3 solutions to be easier to troubleshoot and simpler to maintain. Layer 2 domains can be a real beast, especially when you get broadcast traffic involved. And traffic engineering over a layer 2 link is a real pain in the rear as well.

I am sure we would want a different internet link (with a different provider) for DR purposes as well as load balancing. To me we could easily solve this by getting a bigger primary internet pipe and then a seperate pipe for replication traffic but that could be because of my lack of knowledge of WAN solutions.

well, you need to do some sort of baseline of the traffic you're going to be pushing over the replication link. Until you do that, you have no idea what kind of link you'll need. If the traffic you'll need to push can be handled by a commercial cable or DSL circuit, for example, you could save alot of money simply by ordering that, and then establishing the IPSEC tunnel back to your primary site. That's boatloads cheaper than ordering a dedicated circuit.

And since you're saturating the pipe you have, you need to do some baselining there as well to determine what kind of traffic you're pushing to see if you can either get rid of some of it, or how much of an increase you actually need.

So get some Netflow collection going on and find out what's happening on your network, then you'll be able to make decisions more intelligently.

Bl8ckr0uter

Forsaken_GA wrote: »

So get some Netflow collection going on and find out what's happening on your network, then you'll be able to make decisions more intelligently.

You are absolutely correct. I really need to get on that.

Do you have any suggestions for some light to moderate WAN technology reading? I downloaded some Cisco design pdfs and I plan on picking up some MPLS pdfs as well. I don't think I am ready for any of the dedicated MPLS books yet but I would like a real world resource for WAN technology information.

Forsaken_GA

Bl8ckr0uter wrote: »

You are absolutely correct. I really need to get on that.

Do you have any suggestions for some light to moderate WAN technology reading? I downloaded some Cisco design pdfs and I plan on picking up some MPLS pdfs as well. I don't think I am ready for any of the dedicated MPLS books yet but I would like a real world resource for WAN technology information.

Well, I don't really think this is much of a you need more info on WAN technology kind of issue. Getting down into the nitty gritty details of the tech doesn't really do much to help you. And honestly, for most of the service provider solutions, the customer isn't really involved. The magic all happens on the backend, which you won't have any control over. If you get a layer 2 circuit end to end, as far as your switch is concerned, it's hooked directly to another switch at your other site, and there's no network in between. If you get a layer 3 vpn MPLS solution, all you generally need to do is configure a BGP session to the service provider on both ends, and then you're good to go. I believe you're already familiar with establishing IPSEC tunnels over the public internet.

You need to do some more reading on practical network design and management, and there aren't any really good texts out for those, since there's so many different variables, and design/management strategy is ultimately driven by the needs of the business.

The Network Administrator's Survival Guide is a good place to start, but honestly, the only real way to learn this crap is to get into it, and make your mistakes, tempered with advice from other folks who've made the mistakes before.

The very first thing you need to get a handle on is the traffic flow in your network. If you don't know the types of traffic that are on your network, and the paths they take to move around, you're just flailing in the dark.

Bl8ckr0uter

Forsaken_GA wrote: »

Well, I don't really think this is much of a you need more info on WAN technology kind of issue. Getting down into the nitty gritty details of the tech doesn't really do much to help you. And honestly, for most of the service provider solutions, the customer isn't really involved. The magic all happens on the backend, which you won't have any control over. If you get a layer 2 circuit end to end, as far as your switch is concerned, it's hooked directly to another switch at your other site, and there's no network in between. If you get a layer 3 vpn MPLS solution, all you generally need to do is configure a BGP session to the service provider on both ends, and then you're good to go. I believe you're already familiar with establishing IPSEC tunnels over the public internet.

You need to do some more reading on practical network design and management, and there aren't any really good texts out for those, since there's so many different variables, and design/management strategy is ultimately driven by the needs of the business.

The Network Administrator's Survival Guide is a good place to start, but honestly, the only real way to learn this crap is to get into it, and make your mistakes, tempered with advice from other folks who've made the mistakes before.

The very first thing you need to get a handle on is the traffic flow in your network. If you don't know the types of traffic that are on your network, and the paths they take to move around, you're just flailing in the dark.

I'll pick up the book (8 bucks used on amazon) and take your suggestion. Thanks.

Forsaken_GA

Bl8ckr0uter wrote: »

I'll pick up the book (8 bucks used on amazon) and take your suggestion. Thanks.

No problem

Things like this is also why I heavily advocate that network guys should get good with Unix. There are so many good and free tools that can make getting the information you need available, but you have to know how to use them. There are some really great Windows based tools (We use Solarwinds Orion here, and I love it), but most of them come at a premium, and the bosses generally like it when I do everything I can to keep capex and opex as low as possible

Bl8ckr0uter

Forsaken_GA wrote: »

No problem

Things like this is also why I heavily advocate that network guys should get good with Unix. There are so many good and free tools that can make getting the information you need available, but you have to know how to use them. There are some really great Windows based tools (We use Solarwinds Orion here, and I love it), but most of them come at a premium, and the bosses generally like it when I do everything I can to keep capex and opex as low as possible

Looks like it will be a good read although I am sure it will be a bit dated. I am ordering it in a few minutes.

I really need to study windows/servers from a network engineers perspective. Like how things look on the network, spotting trouble, etc. I think the wireshark book will be valuable in that regard.

How's gsec going?

networker050184

If you are going for any kind of bandwidth intensive replication between sites I'd try to get a pipe that is as dedicated as possible. What ever you go with make sure you get a nice SLA with bandwidth and delay guarantees.

Bl8ckr0uter

networker050184 wrote: »

If you are going for any kind of bandwidth intensive replication between sites I'd try to get a pipe that is as dedicated as possible. What ever you go with make sure you get a nice SLA with bandwidth and delay guarantees.

Which would you go with given the options I gave?

networker050184

Personally, I'd try to get some dedicated fiber run as my first preference. This option can be very expensive and depends on availability in your area though.

Second would be the Metro-E or other type of MPLS circuit.

I definitely wouldn't go with IPSEC tunnels over the internet. You probably aren't going to get much of an SLA on that.

Forsaken_GA

networker050184 wrote: »

Personally, I'd try to get some dedicated fiber run as my first preference. This option can be very expensive and depends on availability in your area though.

Second would be the Metro-E or other type of MPLS circuit.

I definitely wouldn't go with IPSEC tunnels over the internet. You probably aren't going to get much of an SLA on that.

Hehe, virtually none. That's why we use them as a backup option.

For a small company though, getting some dedicated fiber may not be an option, especially since they usually want very long contract times.

And for DB replication, the SLA gives me someone to yell at, but more money is lost if that data gets too far out of sync than I could possibly recoup for an SLA infraction, so redundancy is a *boatload* more important

networker050184

Forsaken_GA wrote: »

Hehe, virtually none. That's why we use them as a backup option.

For a small company though, getting some dedicated fiber may not be an option, especially since they usually want very long contract times.

And for DB replication, the SLA gives me someone to yell at, but more money is lost if that data gets too far out of sync than I could possibly recoup for an SLA infraction, so redundancy is a *boatload* more important

Yeah, but that redundancy doesn't do much for you when your replication fails over the crappy IPSEC tunnel.

Forsaken_GA

networker050184 wrote: »

Yeah, but that redundancy doesn't do much for you when your replication fails over the crappy IPSEC tunnel.

Oh, I agree. As I mentioned, we use them *only* for backup purposes. Unfortunately, our MPLS provider has issues on a rather frequent basis. I am very sick of backward transition notifications at 3am. And unfortunately, I don't have the same level of authority when it comes to dealing with service providers that I used to. Getting a circuit from a different provider run into my facility would take an act of God to get it through the bureaucracy.

But, depending on the amount of traffic he's putting through it, an IPSEC tunnel can work fine. If he only needs to ****, say, 2mb down to a remote site for replication traffic, getting yourself into a 20 year contract to light up some dark fiber is not the most cost effective solution. If he needs something closer to 30mb, and there are frequent bursts during business hours because people are inserting a ton of data, then I completely agree, get a dedicated circuit with some speed and some guarantees on it. That's why my main suggestion is to first get a baseline and see what's actually going on. It doesn't sound like the replication is setup yet, and if I were him, I would set it up on the local LAN so that I could at least get an idea of how much traffic it's going to push. Ordering a circuit without knowing your bandwidth needs is shooting yourself in the foot.

But whatever solution he ends up going with, he's going to need to sell management on the idea of redundancy, because that primary circuit *will* go down every once in awhile, and as I mentioned before, having databases that are hugely out of sync causes some major problems, but I suppose that's a topic for another forum hehe

Forsaken_GA

Bl8ckr0uter wrote: »

Looks like it will be a good read although I am sure it will be a bit dated. I am ordering it in a few minutes.

I really need to study windows/servers from a network engineers perspective. Like how things look on the network, spotting trouble, etc. I think the wireshark book will be valuable in that regard.

How's gsec going?

It is a bit dated. For example, when it gets into Network Monitoring systems, it discusses a very old version of Nagios and Big Brother. I know of no one who actually uses Big Brother anymore, but the idea behind it is to give you a conceptual understanding of how an NMS should work, what you should look for, and that should lead you to look at modern NMS systems.

The CCDA/DP material is somewhat ok for basic theoretical design, but it's weak on specifics. And to put it nicely, Cisco's ideas of network design are what I consider pie in the sky. Best practice can be very tough and will be very expensive to pull off, so you've got to learn to be flexible and leverage what you have to cover your needs, and in quite a few companies, you'll have to have some serious business justification if you need more assets, whether it be equipment, services, or headcount.

From the network perspective, servers are just packet generators. The only thing we really care about is whether the company's needs mean I have to treat certain packets better than others, and whether some of them are the cause of abnormal packet generation.

As an example, one day a company I worked for implemented this pretty new Asset tracking software. The problem was, they had all the servers report in to the centralized management server. At the same time. This was flooding a router interface, and caused that router to miss some OSPF hello's. So the entire OSPF area reconverged. Then it got the next hello, and it reconverged again. Then it missed one, and so on. Every hour on the hour, my ospf routers were going into freakout mode because of this one service, and disrupting all internal communications for about 5 minutes.

So I blocked it, and I yelled at the admins to stagger their reporting, or else I'd be lobbying management to take the money I needed to upgrade the router to handle that traffic out of *their* budget.
Problem went away shortly thereafter. I didn't care about the contents of the packet, that was utterly irrelevant to me. All I cared about is what the packet generators did to my pipes.

It's alot like being a traffic cop. I don't care if you drive a Honda or a Volvo, but I care quite a lot if you stall in the middle of the highway and cause a 90 minute delay (or worse, a 10 car pileup)

Bl8ckr0uter

I guess i'll tack this on to this thread.


For a traffic study is something like a netflow an ok substitute for actual packets or is it best to have the actually pcaps?

Forsaken_GA

Bl8ckr0uter wrote: »

I guess i'll tack this on to this thread.


For a traffic study is something like a netflow an ok substitute for actual packets or is it best to have the actually pcaps?

netflow is going to be your primary source of information. it gives you plenty of details about the flows themselves without needing to capture actual data, and if you use a collector that can graph the results, it'll show you the trends and allow you to find out specifically what flows are going on at what time between what machines and ports, etc. You can run netflow fulltime, and it's impact on resources will be minimal, this allows you to get better trending data. Running a pcap full time is going to be murder on your disk i/o, your storage space, and take forever to actually analyze. I use pcaps as part of the investigative process, when I have an idea of the data I'm looking for, but want to see what's actually crossing the wire

Panzer919

Who is your current provider? I know an ISP in your area that does Q-in-Q tunnels for these kind of applications.

Bl8ckr0uter

Panzer919 wrote: »

Who is your current provider? I know an ISP in your area that does Q-in-Q tunnels for these kind of applications.

TW Telecom

Panzer919

TWCBC does fiber q-in-q tunnels, it would allow you to just have a pipe offsite and on a different network for backups

wolverene13

Panzer919 wrote: »

TWCBC does fiber q-in-q tunnels, it would allow you to just have a pipe offsite and on a different network for backups

Just to clarify for you, Blackrouter, this would be an Ethernet technology. MPLS is typically going to be over T-1's or bonded T-1's. If it's over fiber or copper, even if it's MPLS, it's going to be MPLS over Ethernet anyway. You might as well go with Ethernet. And I'd have to disagree with the assertion that MPLS "just works." MPLS is going to be far more complicated than your typical Ethernet setup (which is normally Q in Q) being that there is much more configuration involved and the Layer 3 aspect is added to the equation. Ethernet Q in Q acts as if your devices are directly connected. It's just a Layer 2 tunnel through the provider's network that connects your sites together. There's no LSP's, VPLS's, MVPLS's, ELAN's, EVPL's or anything like that to get in the way. Remember, the more complicated WAN technology is, the more likely it is to break when someone hoses something up in the ISP's cloud. It happens at least twice a week where I work and sometimes up to 100,000 customers can be out of service.

Forsaken_GA

wolverene13 wrote: »

You might as well go with Ethernet. And I'd have to disagree with the assertion that MPLS "just works." MPLS is going to be far more complicated than your typical Ethernet setup (which is normally Q in Q) being that there is much more configuration involved and the Layer 3 aspect is added to the equation.

From the customer side? The customer usually has very little to do with the MPLS setup. In the case of a layer 3 VPN, there's no special configuration involved on the customer's side of the link, so from that aspect, yes, it just works. The voodoo all happens on the service provider's end.

notgoing2fail

Just read every post here. I really like this thread so I'm posting here just to keep it bumped... LOL..

Not to steer too off course but I've been playing with OpManager and I can't decide if Opmanager is better than Solarwinds? They both seem to have their own fanboys.....

We have some customers that we backup SQL data remotely and are running into BW issues as well because quite frankly, their pipe isn't big enough. I haven't yet sat down to strategize the best technology/cost to pitch to the customer but as data continues to grow it will become a more serious problem....

wolverene13

Forsaken_GA wrote: »

From the customer side? The customer usually has very little to do with the MPLS setup. In the case of a layer 3 VPN, there's no special configuration involved on the customer's side of the link, so from that aspect, yes, it just works. The voodoo all happens on the service provider's end.

Sorry, I should have clarified. I tend to word things from the provider's perpective. What I meant is that if you want reliability, Ethernet is your better bet. There is much more that can go wrong in the provider's cloud with MPLS being that the provider has to configure much more with MPLS. One little misconfiguration on the provider side can easily hose your connectivity to your other sites or the Internet, but with Ethernet, the provider normally just has a Layer 2 network and uses QinQ tunneling to connect everything, so there is less of a chance that a mishap will occur. Both are pretty much equal with respect to amount of configuration on the customer end.

Forsaken_GA

wolverene13 wrote: »

Sorry, I should have clarified. I tend to word things from the provider's perpective. What I meant is that if you want reliability, Ethernet is your better bet. There is much more that can go wrong in the provider's cloud with MPLS being that the provider has to configure much more with MPLS. One little misconfiguration on the provider side can easily hose your connectivity to your other sites or the Internet, but with Ethernet, the provider normally just has a Layer 2 network and uses QinQ tunneling to connect everything, so there is less of a chance that a mishap will occur. Both are pretty much equal with respect to amount of configuration on the customer end.

Ok, right, my point was though that interconnecting two layer 2 domains has it's own ramifications and design issues, particularly if you already have a large layer 2 domain. If you need to do any kind of traffic engineering, your options are pretty limited if all you take is a layer 2 link.

Each type of circuit has their pros and cons, which is why it's important to identify the business needs and choose the right one based on that. Lord knows we all wish it was as simple as 'just get such and such circuit', but it ain't