ASA - Intermittently cant surf but can ping from inside interface

Lizano

So, I have this ASA, every day the site calls and says they are down. When they are down, they can ping public IP address but they cant ping by name, nor can they browse to websites. They also can't browse to IP addresses.

The part that is killing me is that most of the time, they reboot it, sometime twice, and they are now able to surf. The ASA also has VPN connection that does go down when this occur. ICMP has been enable on the outside interface, and while the site reports they are down, I can ping the outsite interface.

Any ideas?

instant000

Lizano wrote: »

So, I have this ASA, every day the site calls and says they are down. When they are down, they can ping public IP address but they cant ping by name, nor can they browse to websites. They also can't browse to IP addresses.

The part that is killing me is that most of the time, they reboot it, sometime twice, and they are now able to surf. The ASA also has VPN connection that does go down when this occur. ICMP has been enable on the outside interface, and while the site reports they are down, I can ping the outsite interface.

Any ideas?

You start off making the problem seem like DNS ... until you reveal they also can't go to websites even via IP address.


1. My first idea is to just take the ASA out of the loop, and see if they experience the problem again, or not. This would be the easiest test. Is it only the http traffic that breaks at this time? For instance, can they still ftp? Can they perform nslookups against remote dns servers? just trying to confirm the extent of the problem.
2. Funny line of thought, but can you verify their licensing?
3. Also, can you verify they don't use an internal proxy for their internet connection? If not, you could try temporarily turning off the inspects for http.
4. The only other thing I can immediately think of is making sure their timestamps are accurate, then enabling syslog. When they say the connection goes down the next time, see if there is any correlation in the log.
Of course, if you already have logs, then just check those.
5. Do they have business class internet? They aren't hitting any provider caps, and/or anything silly like that, are they?



this is what I have off the top of my head.

Hope this helps.

SteveO86

Any interesting logs? maybe remove DNS from the default inspection policy.

I would test with an NSLookup or pinging IP addresses vs names, just to verify it is a DNS issue then troubleshoot from there.

Lizano

Thanks guys.

Unfortunately no log, I just set up logging on the unit today, so hopefully next time it happens I will have logs. My first thought was this was a DNS issue, that is why I had them try to surf to an IP, I was shocked when that didnt work.

Regarding the bandwidth, I have access to the T1 router after the ASA, and there is no overutilization or anything like that. That is a good point though, I need to try HTTPS and FTP traffic and nslookup when this happens, so far I have only tried ICMP and HTTP.

cisco_trooper

What ASA do you have? 5510, 5520? Start monitoring the number of active connections through the ASA.

FW#show conn

There is a maximum number of connections specified on these units. Also, are you NAT'ing all your egress traffic to one external IP address? You need to keep layer 4 implications in mind here since ICMP remains responsive during the reported outage. You MIGHT be running out of ephemeral ports, or you MIGHT be bumping up against the connection limit of the device.

Are the users unable to get to ANY website and IP address? I'd hate to see anyone overlook a far end web application because they forgot to try pinging www.google.com. I know this is probably not the issue, but I've seen people do (..and not do) silly stuff.

Check the logs on the edge devices for link flaps, and check the firewall logs for lots of SYN timeouts, etc. Make sure that end user traffic is making it to the firewall when the outage is reported.

ptilsen

Not a networking guy by trade, but I've seen this behavior on SMB-class ZyWall, Adtran, and Astaro devices. Ping works, http and many other TCP and UDP based protocols don't, including DNS. The only real commonality between these NAT firewalls was that they were A. cheap, SMB-class firewalls and B. broken. I realize ASAs are a different class of device, but anything that uses electricity can and probaby will eventually break. Unless this ASA has had changes that correlated with the problems or has always been this way, I would consider hardware failure.

In a perfect world, you could test this by using an identically configured ASA of the same model. That will quickly tell you if it's hardware or something else.

kmcintosh78

WE just had an issue with our 5520 in relation to VPN. Issue turns out that the version we were running had a known bug.
What Version are you running? Is it the most up to date?
Do you have 2 ASAs online, where you can force the active to standby and see if the the trouble follows?

Lizano

The hardware has been replaced and this continues to happen, I got the device logging to a syslog server but nothing in the logs sheds any light.

instant000

Lizano wrote: »

The hardware has been replaced and this continues to happen, I got the device logging to a syslog server but nothing in the logs sheds any light.

if you already replaced the hardware and that didn't fix it (and you're not allowed to run without the ASA to see if the problem goes away?)

You have a TAC case open?

Have you set up a capture yet?

You can set up a capture with circular-buffer, and then whenever they find they have the problem, look at the capture file, and see if you see anything funny coming across. If you set the buffer up large enough, it should be OK. (this does use resources though, so let the customer know that you're doing this first )

This site does have static IP, right?
And, everything's on the up-and-up with their ISP, right?

Like I originally asked about licensing, and other guy asked about connection limits.

The strange part is that the logging isn't reporting the issue (and if that is true) then it could be a problem with the provider, if not the device.

Did you hard code the speed and duplex settings?

Also, I did a brief search on this issue, one of the links hit on Cisco support forums, and the guy recommended a TAC case and suggested it was a possible memory depletion issue, and asked the customer about upgrading their firmware. The thing is, I think you need to enable some type of out-of-band access to that device, so you can collect information from the device when the outage occurs ... it could very well be something simple that you're not getting since you're probably unable to remote in when the problem occurs, right ... wouldn't be surprised if the customer didn't want anyone remoting into their firewall, but you gotta troubleshoot somehow.

https://supportforums.cisco.com/thread/2119532

Mrock4

Same problem happened to me- almost exactly. Turned out to be a bad port heading from a VSS core via port channel to the router. They were still able to ping throughout the 'outage' but no pages would load. Once I dropped half of the port channel to isolate the cause..it was fine. Likely not the case here, but interesting.

Mrock4

cisco_trooper wrote: »

Check the logs on the edge devices for link flaps, and check the firewall logs for lots of SYN timeouts, etc. Make sure that end user traffic is making it to the firewall when the outage is reported.

Ignore ANYTHING this guy says........

networker050184

Had some very similar issues with a pair of ASAs but in my scenario the VPN stayed up. Ended up being software related.

Have you checked with Cisco to see if this is a known bug?

Lizano

Crypto lifetime mismatch was the fix.

ChooseLife

Lizano wrote: »

Crypto lifetime mismatch was the fix.

Ah, interesting, thanks for sharing. Hope that saves someone's day some time in the future.