Very odd issue - Firewall issue or something else? Really important!

higherho

All, I am having a very odd issue right now. Our developers are trying to gain access to a webpage for particular development purposes. I've given the public range of IP's our network uses to the other company who is hosting this web server so that we can gain access to it.

Well, in the main domain (non development) we can gain access to this website. However, in the one domain that they want to access it in they cannot. At first I wanted to ping the website but I could not I could not even tracert anything either in any of the domains. So I quickly went to my ASA and I saw that on my inside and outside interface ICMP was disabled, once this was renabled I was able to ping in the main non development environment. However, in the other I Cannot. On our firewall for outside and inside interfaces (we have a total of 6, one for VPN, one for inside, one for outside, and one for each test domain) I'm allowing all HTTP and HTTPS traffic through.

Further testing I took notice that whenever I tried to access this website in the test domain it was going through our load balanced nic (I'm trying to access it from a load balanced web server). Once I saw the packets getting torn down through that NIC I thought I would need to create particular access rules in the firewall interface for that domain. However, this did nothing.

Btw in this domain I can access any other website (google, etc) but I just cannot ping or tracert any website. I am confused and I think I might be overlooking something simple. Is it on my end since I can access their website in production domain?


Any help would be greatly appreciated.

deth1k

Are you allowing DNS?

higherho

Update. So I've used Cisco ASDM Packet tracer to see what is causing some issues. I took the source IP address of the website I am trying to go to and the Desktination IP address of the server that cannot get to it. I've done this test on both the inside and outsite interface (the source port on both is 443). When I selected the inside interface it went out all the way to route lookup and appears to be going out. However, when I select the outside interface the packet trace stops at NAT Look up and tells me that the packet was dropped.


In the NAT phase it says "1 access list LoadBalanced_nat_outbound match tcp LoadBalanced XXX.XXX.XXX.XXX (IP stuff =x) outside any eq 443 dynamic translation to pool 1 (public IP of firewall interface)+ [Interface PAT]

translate_hits =297, untranslate_hits =2

higherho

deth1k wrote: »

Are you allowing DNS?

DNS to my knowledge is configured correctly. I cannot connect to the website via ip either.

higherho

So I tested the other enviroments and they all can get to the website, ping and do tracerts.


EDIT

Apparently this is only happening on both of these servers.

higherho

Still nothing. I'm not sure what I'm missing here my mind seems pretty strained atm (been on this issue for like 5 and a half hours now). I think I'm just missing a key point. The two servers this is having issues on are two web servers that are load balanced. One NIC is setup for IN band traffic and the other NIC is setup for internet, etc. I've tried gaining access to the webpage on other servers that are using the load balance network and they all can get to it. ummmm..

higherho

I dont want people to think I'm spamming or anything. Just putting my logic in each post and trying to solve an issue (this somewhat helps my troubleshooting when I look at all I did). So after investigating a bit more I believe the issue is regarding the NLB. It would make sense considering their the only servers that are having this issue and the only servers in a NLB setup.

So I asked the network team at the company if they are running server 2008 (I'm on 2003R2) and in a NLB setup because I've took notice to this issue going around;

Network Load Balancing (NLB) clients cannot connect to the Windows Server 2008 NLB cluster by using the virtual IP address when NLB is running in multicast mode

The only thing though is when I check the firewall when I'm trying to go out to the website its not using the virtual IP of the NLB it actually uses the server IP.

Everyone

Is there a proxy between the firewall and the web servers?

I had a pair of servers setup with NLB once, and this stupid proxy blocked traffic to the virtual IP for the NLB cluster. If I changed the firewall rule to go to either server's IP individually, it worked. If the proxy was turned off, it would work using the virtual IP.

If you can't access it from the virtual IP anywhere, then NLB probably isn't configured properly.

higherho

Everyone wrote: »

Is there a proxy between the firewall and the web servers?

I had a pair of servers setup with NLB once, and this stupid proxy blocked traffic to the virtual IP for the NLB cluster. If I changed the firewall rule to go to either server's IP individually, it worked. If the proxy was turned off, it would work using the virtual IP.

If you can't access it from the virtual IP anywhere, then NLB probably isn't configured properly.

My firewall is not configured to setup as a proxy. I will have to check with the network guys from our other team (our network is segregated from theirs but we use their external connection).

When I try to access the web site I dont see the packets building as the virtual IP but instead an IP that is associated with the web traffic NIC. I just find it extreamly odd that I can access any other web page fine.

Everyone

higherho wrote: »

My firewall is not configured to setup as a proxy. I will have to check with the network guys from our other team (our network is segregated from theirs but we use their external connection).

Right, this proxy was another box that was in-between the firewall and the rest of the network. In my case it was an ISA firewall, and a St. Bernard proxy. I didn't have access to the proxy. Guy who did have access couldn't find any reason for it to block it, it just didn't want to play nice with NLB.

higherho wrote: »

When I try to access the web site I dont see the packets building as the virtual IP but instead an IP that is associated with the web traffic NIC. I just find it extreamly odd that I can access any other web page fine.

If the traffic isn't going to the virtual IP, then NLB isn't going to work.

Each server should have both their on unique IP, and the virtual IP that they share. If you can get the website to come up by going to the unique IP, but not shared virtual IP, than either NLB isn't configured properly, or IIS isn't configured properly. If you can ping the shared virtual IP, then the problem is most likely IIS.

Do the websites happen to be VMs? If they are, and running on different ESX hosts, there's some configuration that has to be done in vSphere for Windows NLB to work between VMs.

higherho

I appreciate your help Everyone!

Everyone wrote: »

Right, this proxy was another box that was in-between the firewall and the rest of the network. In my case it was an ISA firewall, and a St. Bernard proxy. I didn't have access to the proxy. Guy who did have access couldn't find any reason for it to block it, it just didn't want to play nice with NLB.




If the traffic isn't going to the virtual IP, then NLB isn't going to work.

Each server should have both their on unique IP, and the virtual IP that they share. If you can get the website to come up by going to the unique IP, but not shared virtual IP, than either NLB isn't configured properly, or IIS isn't configured properly. If you can ping the shared virtual IP, then the problem is most likely IIS.

Each web server has two nics one connected to a 114 network and the web traffic nic / loadbalancer 116 network. The Virtual IP is configured on each web servers load balance NIC. When you say ping the virtual IP do you mean ping the virtual IP from the IIS boxes themselves? If so I am able to ping that IP on each box. I can also ping the IP from other domains as well.

I've tried going to the website with its unique IP (the web sites IP I assume you are talking about?) and I'm still unable to go to it. I get Internet explorer cannot disaply the webpage.

Do the websites happen to be VMs? If they are, and running on different ESX hosts, there's some configuration that has to be done in vSphere for Windows NLB to work between VMs.

No VM's in this environment.

No the websites are

higherho

Also, the website I am trying to connect to is external to our environment. I just also confirmed that we have no proxy except a blue coat and thats only internal to their network.

Everyone

higherho wrote: »

I appreciate your help Everyone!



Each web server has two nics one connected to a 114 network and the web traffic nic / loadbalancer 116 network. The Virtual IP is configured on each web servers load balance NIC. When you say ping the virtual IP do you mean ping the virtual IP from the IIS boxes themselves? If so I am able to ping that IP on each box. I can also ping the IP from other domains as well.

I've tried going to the website with its unique IP (the web sites IP I assume you are talking about?) and I'm still unable to go to it. I get Internet explorer cannot disaply the webpage.



No VM's in this environment.

No the websites are

The web traffic NIC should have 2 IPs bound to it on each server. Unique IP and shared IP (the virtual IP/ NLB IP). The 2nd (private) NIC should be for heartbeat traffic only, it should not have a gateway configured on it, and it should have only 1 IP bound to it.

In IIS on each server, they need to be configured to respond to requests on the appropriate IP(s) and ports.

A record in DNS for the website should point to the load balanced IP (the one that all these web servers share). So someone trying to browse to www.website.com should connect to the NLB IP. Firewall should be passing traffic to that IP.

If you browse to Server A's public IP (the unique one, not the shared one), as long as IIS is answering on that IP, something should come up. Same for Server B, etc. If nothing comes up, problem is with IIS.

If you can ping the NLB IP (the one that is shared between all of them), but not access the website using that address, then the problem is with IIS. If you can't ping it, then the problem is with NLB (or like I said something weird like a proxy sitting between the firewall and network).

higherho

Everyone wrote: »

The web traffic NIC should have 2 IPs bound to it on each server. Unique IP and shared IP (the virtual IP/ NLB IP). The 2nd (private) NIC should be for heartbeat traffic only, it should not have a gateway configured on it, and it should have only 1 IP bound to it.

Correct on the web traffic nic we actually have multiple IP's one that nic. One is specific to the server and the shared IP is also on that NIC. The only odd thing I see with the web traffic nic is under the "Support" tab address type: Manually configured but the IP listed is an IP associated with one of the internal websites but if I go to the general tab and select Props > scroll down to TCP/IP > Its configured for a different IP to use (which is the IP I see get torn down at the firewall) and no DNS servers are typed in (just has the "use the following DNS server address" selected).

The default gateway is correctly configured on the web traffic NIC.

AS for the inband NIC (heart beat traffic) is configured with no default gateway and has the DNS servers listed.

In IIS on each server, they need to be configured to respond to requests on the appropriate IP(s) and ports.

A record in DNS for the website should point to the load balanced IP (the one that all these web servers share). So someone trying to browse to www.website.com should connect to the NLB IP. Firewall should be passing traffic to that IP.

The domain that the web servers are sitting in that DNS has a host A record (within the forward lookup zone of the domain) with that virtual IP associated to it. THE FQDN of the host A record is the name of the NLB.

If you browse to Server A's public IP (the unique one, not the shared one), as long as IIS is answering on that IP, something should come up. Same for Server B, etc. If nothing comes up, problem is with IIS.

Would IIS block me from visiting an external website though? these servers do not have public IP's associate to them (the websites on the box do though). Any traffic going out to the web goes to the firewall and out through our firewalls external interface with that public IP that is associate to it.

If you can ping the NLB IP (the one that is shared between all of them), but not access the website using that address, then the problem is with IIS. If you can't ping it, then the problem is with NLB (or like I said something weird like a proxy sitting between the firewall and network).

I'm trying to hit someone elses website with these two servers (outside of our network). This isn't about the local websites on the boxes themselves.

higherho

The heat beat NIC on the other web server has multiple IP's on it. Not sure why it does (I did not configure this NLB) so I will need to look into that.

Everyone

Sounds like DNS is configured on the wrong NIC. Heartbeat NIC doesn't need DNS, the other NIC should have that.

higherho wrote:

I'm trying to hit someone elses website with these two servers (outside of our network). This isn't about the local websites on the boxes themselves.

Ok that changes things a lot, I must have misunderstood or overlooked that.

NLB is inbound only. All outbound traffic will come from the unique public IP of each server, it will NEVER come from the NLB IP. So your firewall needs to allow traffic from Server A AND Server B's IP to whatever the destination is, etc.

higherho

Everyone wrote: »

Sounds like DNS is configured on the wrong NIC. Heartbeat NIC doesn't need DNS, the other NIC should have that.


Ok that changes things a lot, I must have misunderstood or overlooked that.

NLB is inbound only. All outbound traffic will come from the unique public IP of each server, it will NEVER come from the NLB IP. So your firewall needs to allow traffic from Server A AND Server B's IP to whatever the destination is, etc.

No problem, its actually helping my understanding a bit more on NLBS so you helped a lot their So on my firewall, I'm allowing all http and https traffic through. The packets are getting natted to the external IP of the firewall and going out. The whole environment (that the load balance network is on) has two access rules - one rule is the implicit rule and the other is an any any rule allowing IP through. The inside interface and outside interface are allowing all http / https traffic through to all domains.

higherho

Wow, so I figured out what was blocking the access. It was the certs associated with the web site. The minute I took out the CA cert I was given and the minute I took out the user cert I was able to hit the website =/ Though of course I cannot get into the site without the user cert.

Everyone

What about the remote end though? Do you know which end initiates the session?