ASA5520 to 5505 L2L VPN

NetstudentNetstudent Member Posts: 1,693 ■■■□□□□□□□
I have an ipsec l2l vpn between a 5520 and a 5505 running 804. The tunnel drops everytime ISAKMP rekeys. The configs are correct, and I have already gotten TAC and several engineers to review the config.

Basically what happens is on the far-end 5505 ISAKMP tears down the tunnel about 300 seconds before the actual timer expires. Timer is set to 3600s for isakmp and ipsec.

When this happens, the ISAKMP SA on the 5505 shows this:
IKE Peer: x.x.x.x
    Type    : user            Role    : initiator 
    Rekey   : no              State   : MM_WAIT_MSG2?
    Encrypt : aes-256 ?        Hash    : SHA       
    Auth    : preshared       Lifetime: 0 ?
5505# "show cry ipsec sa"

There are no ipsec sas


The 5520 at the head end shows this:
1   IKE Peer: x.x.x.x
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_ACTIVE
    Encrypt : 3des            Hash    : SHA
    Auth    : preshared       Lifetime: 3600
    Lifetime Remaining: 339

So the headend is still counting down and the far-end has already tore down the tunnel.

ALso I am running captures on UDP:500 on the outside interface with the peers specified in the ACL on both firewalls. The 5505 sends and sends but the 5520 does not reply.

On the 5505 you can see the ISAKMP packets going out and the 5520 not replying (2.2.2.2 would be the headend):
   61: 12:46:58.337827 1.1.1.1.500 > 2.2.2.2.500:  udp 108
  62: 12:47:12.422265 1.1.1.1.500 > 2.2.2.2.500:  udp 68
  63: 12:47:12.426094 1.1.1.1.500 > 2.2.2.2.500:  udp 84

This goes on for about 5 minutes OR untill I clear the crypto isakmp sa on the headend. When I clear the crytpo or if I wait 5-6 minutes then I will see bidirectional packets coming and going.

It may also be worth nothing that we just upgraded to 804 and the problem started to occur around the same time as the upgrade, but nobody has confirmed that this is a code issue.

WHen I run debugs on the 5505 I see that ISAKMP tries to start a rekey but does not get a reply from the 5520.


<167>Oct 03 2008 14:49:40: %ASA-7-713906: IP = 2.2.2.2, Starting phase 1 rekey

<165>Oct 03 2008 14:49:40: %ASA-5-713041: IP = 2.2.2.2, IKE Initiator: Rekeying Phase 1, Intf outside, IKE Peer 2.2.2.2 local Proxy Address N/A, remote Proxy Address N/A,  Crypto map (N/A)

<167>Oct 03 2008 14:49:40: %ASA-7-715046: IP = 2.2.2.2, constructing ISAKMP SA payload

<167>Oct 03 2008 14:49:40: %ASA-7-715046: IP = 2.2.2.2, constructing Fragmentation VID + extended capabilities payload

<167>Oct 03 2008 14:49:40: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108

<167>Oct 03 2008 14:49:48: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108

<167>Oct 03 2008 14:49:56: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108

<167>Oct 03 2008 14:50:04: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108

<167>Oct 03 2008 14:50:12: %ASA-7-715065: IP = 2.2.2.2, IKE MM Initiator FSM error history (struct &0xd791052[IMG]https://us.v-cdn.net/6030959/uploads/images/smilies/icon_cool.gif[/IMG]  <state>, <event>:  MM_DONE, EV_ERROR-->MM_WAIT_MSG2, EV_RETRY-->MM_WAIT_MSG2, EV_TIMEOUT-->MM_WAIT_MSG2, NullEvent-->MM_SND_MSG1, EV_SND_
MSG-->MM_SND_MSG1, EV_START_TMR-->MM_SND_MSG1, EV_RESEND_MSG-->MM_WAIT_MSG2, EV_RETRY

<167>Oct 03 2008 14:50:12: %ASA-7-713906: IP = 2.2.2.2, IKE SA MM:b3fdfbb2 terminating:  flags 0x01000422, refcnt 0, tuncnt 0

<167>Oct 03 2008 14:50:12: %ASA-7-713906: IP = 2.2.2.2, sending delete/delete with reason message

<163>Oct 03 2008 14:50:12: %ASA-3-713902: IP = 2.2.2.2, Removing peer from peer table failed, no match!

<164>Oct 03 2008 14:50:12: %ASA-4-713903: IP = 2.2.2.2, Error: Unable to remove PeerTblEntry

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Group = 2.2.2.2, IP = 2.2.2.2, Expired IKE SA with 1 phase 2 centries still associated

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Group = 2.2.2.2, IP = 2.2.2.2, sending delete/delete with reason message

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing blank hash payload

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing IPSec delete payload

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing qm hash payload

<167>Oct 03 2008 14:50:17: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE SENDING Message (msgid=170945d) with payloads : HDR + HASH ([IMG]https://us.v-cdn.net/6030959/uploads/images/smilies/icon_cool.gif[/IMG] + DELETE (12) + NONE (0) total length : 68

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Group = 2.2.2.2, IP = 2.2.2.2, Active unit receives a delete event for remote peer 2.2.2.2.



<167>Oct 03 2008 14:50:17: %ASA-7-715009: Group = 2.2.2.2, IP = 2.2.2.2, IKE Deleting SA: Remote Proxy 0.0.0.0, Local Proxy 10.10.0.0

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Group = 2.2.2.2, IP = 2.2.2.2, IKE SA MM:63bba120 terminating:  flags 0x0120c002, refcnt 0, tuncnt 0

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Group = 2.2.2.2, IP = 2.2.2.2, sending delete/delete with reason message

<166>Oct 03 2008 14:50:17: %ASA-6-602304: IPSEC: An inbound LAN-to-LAN SA (SPI= 0x78B07B12) between 1.1.1.1 and 2.2.2.2(user= 2.2.2.2) has been deleted.

<166>Oct 03 2008 14:50:17: %ASA-6-602304: IPSEC: An outbound LAN-to-LAN SA (SPI= 0xB4FA3DE6) between 1.1.1.1 and 2,2,2,2 (user= 2.2.2.2) has been deleted.

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing blank hash payload

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing IKE delete payload

<167>Oct 03 2008 14:50:17: %ASA-7-715046: Group = 2.2.2.2, IP = 2.2.2.2, constructing qm hash payload

<167>Oct 03 2008 14:50:17: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE SENDING Message (msgid=1a400590) with payloads : HDR + HASH ([IMG]https://us.v-cdn.net/6030959/uploads/images/smilies/icon_cool.gif[/IMG] + DELETE (12) + NONE (0) total length : 80

<167>Oct 03 2008 14:50:17: %ASA-7-715077: Pitcher: received key delete msg, spi 0x78b07b12

<167>Oct 03 2008 14:50:17: %ASA-7-715077: Pitcher: received key delete msg, spi 0x78b07b12

<167>Oct 03 2008 14:50:17: %ASA-7-715077: Pitcher: received key delete msg, spi 0xb4fa3de6

<163>Oct 03 2008 14:50:17: %ASA-3-713902: Group = 2.2.2.2, IP = 2.2.2.2, Removing peer from peer table failed, no match!

<164>Oct 03 2008 14:50:17: %ASA-4-713903: Group = 2.2.2.2, IP = 2.2.2.2, Error: Unable to remove PeerTblEntry

<164>Oct 03 2008 14:50:17: %ASA-4-113019: Group = 2.2.2.2 Username = 2.2.2.2, IP = 2.2.2.2, Session disconnected. Session Type: IPsec, Duration: 0h:51m:37s, Bytes xmt: 84531039, Bytes rcv: 255159181, Reason: User Requested

<167>Oct 03 2008 14:50:17: %ASA-7-713906: Ignoring msg to mark SA with dsID 507904 dead because SA deleted

<167>Oct 03 2008 14:50:17: %ASA-7-715077: Pitcher: received a key acquire message, spi 0x0

<165>Oct 03 2008 14:50:17: %ASA-5-713041: IP = 2.2.2.2, IKE Initiator: New Phase 1, Intf inside, IKE Peer 2.2.2.2 local Proxy Address 10.10.0.0, remote Proxy Address 0.0.0.0,  Crypto map (outside_map)

<167>Oct 03 2008 14:50:17: %ASA-7-715046: IP = 2.2.2.2, constructing ISAKMP SA payload

<167>Oct 03 2008 14:50:17: %ASA-7-715046: IP = 2.2.2.2, constructing Fragmentation VID + extended capabilities payload

<167>Oct 03 2008 14:50:17: %ASA-7-713236: IP = 2.2.2.2, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108

<167>Oct 03 2008 14:50:17: %ASA-7-715077: Pitcher: received a key acquire message, spi 0x0

<166>Oct 03 2008 14:50:17: %ASA-6-713219: IP = 2.2.2.2, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.

<167>Oct 03 2008 14:50:18: %ASA-7-715077: Pitcher: received a key acquire message, spi 0x0

<166>Oct 03 2008 14:50:18: %ASA-6-713219: IP = 2.2.2.2, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.

<167>Oct 03 2008 14:50:19: %ASA-7-715077: Pitcher: received a key acquire message, spi 0x0

<166>Oct 03 2008 14:50:19: %ASA-6-713219: IP = 2.2.2.2, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.

<167>Oct 03 2008 14:50:19: %ASA-7-715077: Pitcher: received a key acquire message, spi 0x0

<166>Oct 03 2008 14:50:19: %ASA-6-713219: IP = 2.2.2.2, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.

Here is a snippet of debug. The KEY_ACUIRE messages continue while the 5505 sends a ISAKMP rekey payload but the 5520 does not reply.

Again the tunnel runs fine untill a rekey. And the tunnel will finally come back up if I let it sit there long enough. Also the routing is fine, its routing over the internet.

So has anyone encountered this type of behavior?
There is no place like 127.0.0.1 BUT 209.62.5.3 is my 127.0.0.1 away from 127.0.0.1!

Comments

  • networker050184networker050184 Mod Posts: 11,962 Mod
    I haven't seen this before, but if it started happening only after a code upgrade I'd revert back and see if it fixes the issue. If it does you might be missing something in the config that is different between the version. Or it could be a bug...



    Nice to see you around Netstudent :D
    An expert is a man who has made all the mistakes which can be made.
  • dtlokeedtlokee Member Posts: 2,378 ■■■■□□□□□□
    I would change the timers fir IKE and IPSEC so that they are not trying to re-key at the same time, are you using PFS or DPD?
    The only easy day was yesterday!
  • NetstudentNetstudent Member Posts: 1,693 ■■■□□□□□□□
    Hey thanks for the reply DT. I tried several different variations on isakmp and ipsec timers in an attempt to find a stable config. None worked. Also had isakmp keepalives enabled on both sides. I just did a zero downtime code change on the HA pair . Went from 8.0.4 to 8.0.4(7) which is an interim release not available to the public yet. Sure enough, that fixed it. I even made a few crypto map changes and the tunnel only dropped like 4 or 5 ping packets during the change. If I would have done that before, I would have been clearing cryptos like a mad man trying to get it back up.


    Nice to see you around Netstudent
    Thanks, I'm still around. Just lurking lately. I'm trying to get done with school. I'm so burned out with school, and I'm trying to concentrate on finishing without slacking off my grades. The end is hard. I am starting the ASA specialist track though because of work. I think I can get that done real quick. I have enough learning credits for the SNAA course and the SNAf topics I do on a daily basis. Not trying to go Security Pro or anything yet, but perimeter security should be in every engineers toolkit.
    There is no place like 127.0.0.1 BUT 209.62.5.3 is my 127.0.0.1 away from 127.0.0.1!
  • networker050184networker050184 Mod Posts: 11,962 Mod
    Good luck with school and hopefully you will get a little down time for those certs!
    An expert is a man who has made all the mistakes which can be made.
  • AhriakinAhriakin Member Posts: 1,799 ■■■■■■■■□□
    Have you tried keying the lifetime off of Data sent instead of absolute time? It's possible one unit has a clock issue, unlikely but worth seeing if you can isolate it to the function it is using to decide it's time to rekey the SA.
    We responded to the Year 2000 issue with "Y2K" solutions...isn't this the kind of thinking that got us into trouble in the first place?
  • techistoretechistore Member Posts: 1 ■□□□□□□□□□
    Hi Netstudent,

    We are in to the exactly same scenario. We have same hardware 5505 and 5520 and trying to acheive the same LAN-to-LAN connections. We currently have a case open with Cisco and we referred this thread to them so they will give us the code which fixed your issue. Apparently they are asking us to disable idle timeouts in the group policy settings and then see if it is fixing the issue.

    Can you please check if the new code you have has the idle time out disabled? If not can you share the code with us then...
  • cisco_troopercisco_trooper Member Posts: 1,441 ■■■■□□□□□□
    man, i'm glad to see this topic. i was just pondering upgrading my ASAs to the 804 code. With more than one Site To Site VPN in production, perhaps I'll wait until I hear more about this....
  • NetstudentNetstudent Member Posts: 1,693 ■■■□□□□□□□
    techistore wrote:
    Hi Netstudent,

    We are in to the exactly same scenario. We have same hardware 5505 and 5520 and trying to acheive the same LAN-to-LAN connections. We currently have a case open with Cisco and we referred this thread to them so they will give us the code which fixed your issue. Apparently they are asking us to disable idle timeouts in the group policy settings and then see if it is fixing the issue.

    Can you please check if the new code you have has the idle time out disabled? If not can you share the code with us then...

    With the new 804 code, the vpn-idle-timeout was set to 30. TAC never told me that the idle timeout was an issue and I never turned it off. They did mention keepalives and DPD apckets which I made sure was on. But I will add that when all this was happening, I did not see consistent DPD R-U-THERE packets going back and forth between the peers. AFter the change to the interim version, it was going back forth between the peers about once a minute consistently.

    But TAC never gave me a difinitive answer, they were like "Try this didn't work? okay try this, didn;t work okay try this. Until finally they gave me the dang code.

    Even with the interim version, the idle timouts are still on and none of my tunnels are failing.

    If your in a tight spot and need to stabilize and TAC isn't giving you what you need, then PM me.
    Humdinger# show run all group-policy
    group-policy DfltGrpPolicy internal
    group-policy DfltGrpPolicy attributes
     banner none
     wins-server none
     dns-server none
     dhcp-network-scope none
     vpn-access-hours none
     vpn-simultaneous-logins 3
     [b]vpn-idle-timeout 30[/b]
     vpn-session-timeout none
     vpn-filter none
     ipv6-vpn-filter none
     vpn-tunnel-protocol IPSec
     password-storage disable
     ip-comp disable
     re-xauth disable
     group-lock none
     pfs disable
     ipsec-udp disable
     ipsec-udp-port 10000
     split-tunnel-policy tunnelall
     split-tunnel-network-list none
     default-domain none
     split-dns none
     intercept-dhcp 255.255.255.255 disable
     secure-unit-authentication disable
     user-authentication disable
     user-authentication-idle-timeout 30
     ip-phone-bypass disable
     leap-bypass disable
     nem disable
     backup-servers keep-client-config
     msie-proxy server none
     msie-proxy method no-modify
     msie-proxy except-list none
     msie-proxy local-bypass disable
     msie-proxy pac-url none
     vlan none
     nac-settings none
     address-pools none
     ipv6-address-pools none
     smartcard-removal-disconnect enable
     client-firewall none
     client-access-rule none
     webvpn
      url-list none
      filter none
      homepage none
      html-content-filter none
      port-forward name Application Access
      port-forward disable
      mapi disable
      http-proxy disable
      sso-server none
      svc dtls enable
      svc mtu 1406
      svc keep-installer installed
      svc keepalive 20
      svc rekey time none
      svc rekey method none
      svc dpd-interval client 30
      svc dpd-interval gateway 30
      svc compression deflate
      svc modules none
      svc profiles none
      svc ask none
      ike-retry-timeout 10
      ike-retry-count 3
      customization none
      keep-alive-ignore 4
      http-comp gzip
      download-max-size 2147483647
      upload-max-size 2147483647
      post-max-size 2147483647
      user-storage none
      storage-objects value cookies,credentials
      storage-key none
      hidden-shares none
      smart-tunnel disable
      activex-relay enable
      unix-auth-uid 65534
      unix-auth-gid 65534
      file-entry enable
      file-browsing enable
      url-entry enable
    
    There is no place like 127.0.0.1 BUT 209.62.5.3 is my 127.0.0.1 away from 127.0.0.1!
  • cisco_troopercisco_trooper Member Posts: 1,441 ■■■■□□□□□□
    Just got bit by this. icon_mad.gif
Sign In or Register to comment.