Pfsense anyone?

Bl8ckr0uter

[FONT=Arial, sans-serif]Greetings,

I am using 1.2.3 (and have been for a few months). So far it has been rock solid and is exactly what my business has needed. We recently added an additional WAN connection to our existing set up. I added an opt interface (followed instruction in the Pfsense book) created a load balance pool and change my default gateway on my default lan rule. I now have an interesting problem. I did a speed test and my upload speed is now 10mb (the speed of the new wan) and my download speed is 4mb (the speed of my old wan). The first thing I noticed was the fact that it wasn't 14mb like I was expecting it to be but the second thing I noticed was that it obviously isn't using the same wans for upload and download which leads me to believe there is a routing problem. I check and rechecked my config but it seems correct. So now I have a few questions:

1: Do I have to enabling NAT on this device to make load balancing work?
- Natting is done by another device on our network (behind the pfsense). I can change this if it is required but I would like to know.

2: How do you change the default route?
- It would seem that no matter what I do it stays the original wan interface

Anyone have any ideas?
[/FONT]

ehnde

If no one can help you here, they have an IRC channel on freenode -- ##pfsense

You can probably get a good answer to your question in there.

Forsaken_GA

Bl8ckr0uter wrote: »

[FONT=Arial, sans-serif]Greetings,
1: Do I have to enabling NAT on this device to make load balancing work?
- Natting is done by another device on our network (behind the pfsense). I can change this if it is required but I would like to know.

No, you don't need to use NAT to do outbound load balancing.

2: How do you change the default route?
- It would seem that no matter what I do it stays the original wan interface

Anyone have any ideas?
[/FONT]

You have to define other gateways under the routing setup. This will allow you to change the default gateway used wherever a gateway is selectable.

If this is not in production yet, I recommend using 2.0 RC1 instead, btw.

Bl8ckr0uter

Forsaken_GA wrote: »

No, you don't need to use NAT to do outbound load balancing.



You have to define other gateways under the routing setup. This will allow you to change the default gateway used wherever a gateway is selectable.

If this is not in production yet, I recommend using 2.0 RC1 instead, btw.

That's the thing, I don't see where I could do this. Short of putting a 0.0.0.0/1 route, there is no way to actually change the default gateway.

I do want to upgrade this box to 2.0 but is is production already.

Forsaken_GA

You're actually going to make me install a pfense 1.2.3 VM, aren't you, you bastard?

Forsaken_GA

When you configured your load balancing pool, did you set it's type to Server or did you set it to Gateway?

If you set it to server, set it to Gateway, set the monitor IP to whatever is appropriate, then add your two WAN interfaces to the pool, and save it.

Then in any place where you can set a gateway (eg, rules), you can select whatever you named your gateway load balanced pool as the gateway instead of default.

Bl8ckr0uter

Forsaken_GA wrote: »

You're actually going to make me install a pfense 1.2.3 VM, aren't you, you bastard?

Lol well I wouldn't say make, I would say that it would be very helpful to have a Unix expert take a second look at this (hell yea I'm laying it on thick) lol.

Forsaken_GA wrote: »

When you configured your load balancing pool, did you set it's type to Server or did you set it to Gateway?

Here is my topology:

<Internet1> - 4mb <Internet2>- 10bm
| |
<Pfsense>
| |
<InternalFW> - Natting

Keep in mind that all of my natting is done on the internal firewall. The pfsense box is only used for country blocking and for static routes to my two providers.

I set it to Gateway (which according to the pfsense book is the correct way). Server was for, well Web gardens and such. The load balancing pool shows up as green when I check services and the thing is when I do a tracert from a machine on the network, it does take the path of the wan that I want it to. I did the "ghetto" way of unequal load balancing by adding the faster wan's interface in there several times. It is really strange that upload works but download doesn't.

My default "any any" lan rule is pointing at "Wan Load Balance" (my pool name) for it's gateway and like I said, the only problem I have is download, not upload. That makes me believe that it is something with my wan rules coming back, like somewhere in my routing table, I should add the "Wan Load Balance" pool instead of my default gateway (which on 1.2.3 apparently cannot be changed, period).

I did some reading on asymmetrical routing and for the life of me I can't think of a reason A: Why you would want to do this and B: Why this is happening. Check out the pics:

Speed Test:
http://www.speedtest.net/result/1328906339.png


Oh yea I guess I should mention the things I have tried:

1: Writing a firewall rule back from my Wan and Opt interfaces using the Wan LB as the gateway
2: Rewriting all of my firewall rules to using the Wan LB as the gateway (on my lan, wan and opt interfaces)
3: Adding static routes for my Wan LB
4: Changing the default gateway to Wan LB (failed)

Forsaken_GA

Ok, so your problem is that you're trying to load balance ingress traffic, not egress correct? Your load balancer setup is only going to load balance egress traffic, it has absolutely no influence on ingress.

Three very important questions for that then -

#1 - Are the connections with the same provider

#2 - Are you running BGP with your provider(s)?

#3 - Do you have your own IP space that you're announcing to the world, or are you sponging off a provider's IP space?

Long story short, unless you're running multihomed BGP, you're probably screwed. The return traffic is always going to take the best path it finds from it's perspective to come back to you, and that may not always be the link you send it over on. You're now crossing over into the realm of WAN based policy routing, and if all you're doing is taking a couple of circuits and a default route from each provider, you have little to no recourse as to how traffic enters your network.

Bl8ckr0uter

Forsaken_GA wrote: »

Ok, so your problem is that you're trying to load balance ingress traffic, not egress correct? Your load balancer setup is only going to load balance egress traffic, it has absolutely no influence on ingress.

Let me start off by saying thanks because that was literally my next question (does this work for ingress traffic).

Forsaken_GA wrote: »

Three very important questions for that then -

#1 - Are the connections with the same provider

No one is TWTelecom and one is TWCable.

Forsaken_GA wrote: »

#2 - Are you running BGP with your provider(s)?

No. They offered to if I requested but I just haven't done so. Would it be better to just use BGP? Honestly I know very little about BGP ( I am always down for learning something) but it shouldn't be that bad to set up (right?) lol.

Forsaken_GA wrote: »

#3 - Do you have your own IP space that you're announcing to the world, or are you sponging off a provider's IP space?

We do not own our own address space

Forsaken_GA wrote: »

Long story short, unless you're running multihomed BGP, you're probably screwed. The return traffic is always going to take the best path it finds from it's perspective to come back to you, and that may not always be the link you send it over on. You're now crossing over into the realm of WAN based policy routing, and if all you're doing is taking a couple of circuits and a default route from each provider, you have little to no recourse as to how traffic enters your network.

Sigh and I thought this was going to be easy. Ok. Now I have some reading to do. Guess I better get reading on this: http://www.openbgpd.org/

Could I do this over OSPF?

And here is another thought. Why isn't my upload speed 14mb (instead of about 10). I thought the load balancer did bandwidth aggregation....

Forsaken_GA

Bl8ckr0uter wrote: »

No one is TWTelecom and one is TWCable.

Well, different providers makes BGP a requirement, because getting them to both work together to load balance your ingress traffic is doubtful.

No. They offered to if I requested but I just haven't done so. Would it be better to just use BGP? Honestly I know very little about BGP ( I am always down for learning something) but it shouldn't be that bad to set up (right?) lol.

If it were me? Hell yes. BGP gives me the most control over traffic in my network when multihomed to different providers.

For you? Up to you. Traffic engineering is not a simple subject, and badly behaving BGP peers tend to get their sessions shutdown, so I wouldn't recommend implementing BGP with your providers until you're comfortable enough to do that.

We do not own our own address space

This is another problem. If you were to do BGP with both providers, it's unlikely that the provider you aren't leasing space from is going to work with the other to punch holes in their routing policy for you. If you're leasing IP space from both of them, then they'd both need to do the same thing. Unless you're putting out a serious amount of traffic, or represent a serious amount of eyeballs, don't hold your breath (and with Time Warner, don't hold your breath anyway, they're almost as dickish as Level3)

Sigh and I thought this was going to be easy. Ok. Now I have some reading to do.

Could I do this over OSPF?

No. It's not an internal routing problem. This is where you need to understand the concept of routing between autonomous systems. Once the traffic leaves your routing domain, you don't have any say about how it gets to it's final destination anymore.

For example -

Lets say you send traffic out via TWC, and it has connections through Cogent and Level 3 to your final destination. The Level3 link has less latency, so you would prefer it go over that. But it goes over Cogent. Why? Because someone on TWC's side told it to. This is policy based routing, where routing decisions are not always made by the most optimal path. There is absolutely nothing you can do about it - the general rule among network operators is my network, my rules, and you don't tell another operator how to eat their lunch. You can request your provider alter the traffic profile to take the links you prefer. Whether they actually do it....

In the same vein, the same site you're trying to reach responds to you. You'd prefer that traffic return to you via your TWC link, but the provider(s) it's connected to determines their preferred path is through your TWT link. That's the link it's going to come in over. The only possible way for you to influence your inbound traffic is to manipulate BGP attributes, since BGP is the protocol all the providers run between themselves, that's the only method that the end customer has to imprint their wishes on policy routing decisions.

You did not pick an easy problem to try and solve hehe

Forsaken_GA

Bl8ckr0uter wrote: »

And here is another thought. Why isn't my upload speed 14mb (instead of about 10). I thought the load balancer did bandwidth aggregation....

Load balancing is not the same a link aggregation. I believe pfsense does it's load balancing in round robin fashion, and the sessions are sticky, so whatever link it was sent out, is the link that will be used for the life of the session.

Honestly, what is your end goal here? Did you just outgrow your 4 meg pipe and upgraded to a 10 meg and want to use both? Unless you have an actual need for 2 links, you'd be better off just ordering a bigger pipe than trying to aggregate 2 separate links. You need to understand that you can't look at your two links as having an aggregate bandwidth of 14 megs. You have 2 links, one is 10 megs, and one is 4 megs, and your traffic is more or less limited to those speeds depending on which link it goes out. Personally, if I was in your situation, I would leave the 4 meg link alone and relegate it as a backup link in case the primary went down, unless I had very good reasons otherwise.

Bl8ckr0uter

"Dammit Bill!"

Forsaken_GA wrote: »

Well, different providers makes BGP a requirement, because getting them to both work together to load balance your ingress traffic is doubtful.

If it were me? Hell yes. BGP gives me the most control over traffic in my network when multihomed to different providers.

For you? Up to you. Traffic engineering is not a simple subject, and badly behaving BGP peers tend to get their sessions shutdown, so I wouldn't recommend implementing BGP with your providers until you're comfortable enough to do that.

I can get comfortable really quick if I have to lol. I mean it is over my head right now but I have the summer off from school so

Forsaken_GA wrote: »

This is another problem. If you were to do BGP with both providers, it's unlikely that the provider you aren't leasing space from is going to work with the other to punch holes in their routing policy for you. If you're leasing IP space from both of them, then they'd both need to do the same thing. Unless you're putting out a serious amount of traffic, or represent a serious amount of eyeballs, don't hold your breath (and with Time Warner, don't hold your breath anyway, they're almost as dickish as Level3)

No. It's not an internal routing problem. This is where you need to understand the concept of routing between autonomous systems. Once the traffic leaves your routing domain, you don't have any say about how it gets to it's final destination anymore.

For example -

Lets say you send traffic out via TWC, and it has connections through Cogent and Level 3 to your final destination. The Level3 link has less latency, so you would prefer it go over that. But it goes over Cogent. Why? Because someone on TWC's side told it to. This is policy based routing, where routing decisions are not always made by the most optimal path. There is absolutely nothing you can do about it - the general rule among network operators is my network, my rules, and you don't tell another operator how to eat their lunch. You can request your provider alter the traffic profile to take the links you prefer. Whether they actually do it....

In the same vein, the same site you're trying to reach responds to you. You'd prefer that traffic return to you via your TWC link, but the provider(s) it's connected to determines their preferred path is through your TWT link. That's the link it's going to come in over. The only possible way for you to influence your inbound traffic is to manipulate BGP attributes, since BGP is the protocol all the providers run between themselves, that's the only method that the end customer has to imprint their wishes on policy routing decisions.

You did not pick an easy problem to try and solve hehe

Ok. So let's say I was even thinking about doing this. Would I need to go to arin and get ip addresses and an AS? Then what, run openbgp on pfsense and go nuts? I am only asking because I want to know if this will be worth it in the end? The previous network admin only used static routes and the other guy here is basically going to say "Why is it worth it?" I can make an argument for the worth but the feasibility is what I need to know about... I have a few routers at home and a pfsense box or two, I could mock this up using a non routeable AS? Sigh man this just sucks, I had plans to rest this weekend to lol.

Forsaken_GA wrote: »

Load balancing is not the same a link aggregation. I believe pfsense does it's load balancing in round robin fashion, and the sessions are sticky, so whatever link it was sent out, is the link that will be used for the life of the session.

I just reread the page on Link Aggregation and you are correct.

Forsaken_GA wrote: »

Honestly, what is your end goal here? Did you just outgrow your 4 meg pipe and upgraded to a 10 meg and want to use both? Unless you have an actual need for 2 links, you'd be better off just ordering a bigger pipe than trying to aggregate 2 separate links. You need to understand that you can't look at your two links as having an aggregate bandwidth of 14 megs. You have 2 links, one is 10 megs, and one is 4 megs, and your traffic is more or less limited to those speeds depending on which link it goes out. Personally, if I was in your situation, I would leave the 4 meg link alone and relegate it as a backup link in case the primary went down, unless I had very good reasons otherwise.

The funny thing is the 4mb pipe is about to be upgraded to a 10mb pipe later this month. My end goal involves failover (which I know doesn't require load balancing) and load balancing. Honestly I really just need to make the internet faster (and since I cannot ban all of the internet radio/sites/etc sites, the solution is to use the faster pipe). I think you might be right though (about just leaving the backup alone) . But at some point, we have to have more than just static routes.

Forsaken_GA

Bl8ckr0uter wrote: »

I can get comfortable really quick if I have to lol. I mean it is over my head right now but I have the summer off from school so

BGP is not a simple subject. I would highly recommend Sam Halabi's Internet Routing Architectures and Jeff Doyle's Routing TCP/IP 2 if you're interested in pursuing it. There is a very large difference between BGP and your internal routing protocols.

Ok. So let's say I was even thinking about doing this. Would I need to go to arin and get ip addresses and an AS? Then what, run openbgp on pfsense and go nuts? I am only asking because I want to know if this will be worth it in the end? The previous network admin only used static routes and the other guy here is basically going to say "Why is it worth it?" I can make an argument for the worth but the feasibility is what I need to know about... I have a few routers at home and a pfsense box or two, I could mock this up using a non routeable AS? Sigh man this just sucks, I had plans to rest this weekend to lol.

Well, whether or not it's worth it in the end is up to you and your company. Yes, if you're going to do multihomed BGP, you'd need to request a ASN and IP Space from ARIN. And sure, you can mock up BGP implementations in a lab, plenty of Cisco students do it every day. Only real way to learn it. If you're going to multihome and traffic engineer, you'll want full routes from each provider, so you'll need to make sure you have a box beefy enough to handle it.

The funny thing is the 4mb pipe is about to be upgraded to a 10mb pipe later this month. My end goal involves failover (which I know doesn't require load balancing) and load balancing. Honestly I really just need to make the internet faster (and since I cannot ban all of the internet radio/sites/etc sites, the solution is to use the faster pipe). I think you might be right though (about just leaving the backup alone) . But at some point, we have to have more than just static routes.

Well, making the internet faster is a little dubios. What parts of it are slow? You may just want to implement QoS to give preferential treatment for the traffic that matters. That comes at the cost of treating traffic which doesn't matter badly, however.

Otherwise, yeah, the only solution is to throw more pipe at it. That's the one only guaranteed fix.

I'm pretty sure at some point I told you to go implement netflow and start figuring out what traffic is actually going across your network. I'd suggest doing that, and identifying the problem areas if people are complaining about the internet being slow. Then you can tell them exactly *why* the internet is slow. "Well, you see, Julie has been watching Netflix on company time while simultaneously downloading pirated software from newsgroups. If we stopped her from doing that, the network wouldn't appear to be so slow!"

Don't be a *****, gear yourself up with some good information and go fight back!

Bl8ckr0uter

Forsaken_GA wrote: »

BGP is not a simple subject. I would highly recommend Sam Halabi's Internet Routing Architectures and Jeff Doyle's Routing TCP/IP 2 if you're interested in pursuing it. There is a very large difference between BGP and your internal routing protocols.

Will do.

Forsaken_GA wrote: »

Well, whether or not it's worth it in the end is up to you and your company. Yes, if you're going to do multihomed BGP, you'd need to request a ASN and IP Space from ARIN. And sure, you can mock up BGP implementations in a lab, plenty of Cisco students do it every day. Only real way to learn it. If you're going to multihome and traffic engineer, you'll want full routes from each provider, so you'll need to make sure you have a box beefy enough to handle it.

Well, making the internet faster is a little dubios. What parts of it are slow? You may just want to implement QoS to give preferential treatment for the traffic that matters. That comes at the cost of treating traffic which doesn't matter badly, however.

Otherwise, yeah, the only solution is to throw more pipe at it. That's the one only guaranteed fix.

I'm pretty sure at some point I told you to go implement netflow and start figuring out what traffic is actually going across your network. I'd suggest doing that, and identifying the problem areas if people are complaining about the internet being slow. Then you can tell them exactly *why* the internet is slow. "Well, you see, Julie has been watching Netflix on company time while simultaneously downloading pirated software from newsgroups. If we stopped her from doing that, the network wouldn't appear to be so slow!"

I did I followed your advice man. It didn't go over very well (actually it didn't go over at all). It's hard to say 'Hey boss could you stop uploading gigs of data to dropbox and listening to pandora and watching videos all the time. Make sure you tell everyone else too lol.

Well you have given me good info. Time to get to and figure out what to do next.