Help please

joshgibson82joshgibson82 Member Posts: 80 ■■□□□□□□□□
in CCNP
I'm coming to this board because I know there is a wide variety of experience in here. I hope someone can help me solve a mystery in my network. I have an 1811 router connected to a metro ethernet connection. It has 3Mbps symmetrical bandwidth. When I get to speedtest.net and do a speed test, the download starts off at around 3 Meg, then quickly drops to maybe 13 Kbps. When I try to download files from certain websites (wireshark.org for example), the same thing happens. The download will start out around 400 KBps, then drop to around 5KBps. I have taken a network trace and do not see any problems. I can post that trace if someone would like to see it. I am using CBAC on the 1811 and suspect there may be a software bug with it. I have taken a laptop directly to the metro ethernet switch and connected, downloaded wireshark and other files very quickly. I am posting my Firewall config:

ip inspect log drop-pkt
ip inspect name FW_OUT cuseeme
ip inspect name FW_OUT dns
ip inspect name FW_OUT ftp
ip inspect name FW_OUT https
ip inspect name FW_OUT icmp
ip inspect name FW_OUT tftp
ip inspect name FW_OUT tcp
ip inspect name FW_OUT udp
ip inspect name FW_OUT pptp
ip inspect name FW_OUT http
ip inspect name FW_OUT h323
ip inspect name FW_OUT imap
ip inspect name FW_OUT pop3
ip inspect name FW_OUT rcmd
ip inspect name FW_OUT realaudio
ip inspect name FW_OUT rtsp
ip inspect name FW_OUT esmtp
ip inspect name FW_OUT sqlnet
ip inspect name FW_OUT streamworks
ip inspect name FW_OUT vdolive
ip inspect name FW_OUT telnet
ip inspect name FW_OUT ssh
ip ips notify SDEE

interface FastEthernet1
description Internet Connection
ip address x.x.x.x 255.255.255.192
ip access-group FW_IN in
no ip redirects
no ip proxy-arp
ip nat outside
ip inspect FW_OUT out
ip virtual-reassembly
ip tcp adjust-mss 1460
speed 100
full-duplex
no cdp enable
crypto map TO-JOSH

(the metro E switch is manually set to 100/full duplex and has an MTU of 1518 )

interface Vlan1
description $FW_INSIDE$
ip address 192.168.10.2 255.255.255.0
no ip redirects
no ip proxy-arp
ip nat inside
ip virtual-reassembly
ip tcp adjust-mss 1452

ip nat inside source route-map NAT_POOL interface FastEthernet1 overload

access-list 100 permit ip 192.168.10.0 0.0.0.255 any
route-map NAT_POOL permit 1
match ip address 100

ip access-list extended FW_IN
permit icmp any host <F1 ip address>
deny ip any any log



TOPOLOGY:

(192.168.10.0 /24 segment is VLAN 1) ----> 1811 interface F1 for WAN
> Metro Ethernet switch
> Internet

Again, most all internet sessions work fine, except when I try to do large downloads it seems. I have played with the tcp adjust-mss size and also the mtu on the F1 interface but nothing has made any difference. PLEASE HELP!!!!!!!!!!! It's driving me crazy!
Josh, CCNP CWNA
· Share on FacebookShare on Twitter

Comments

  • joshgibson82joshgibson82 Member Posts: 80 ■■□□□□□□□□
    I'd be more than happy to run any debug commands someone would want to help troubleshoot this. I'm just out of ideas at this point.
    Josh, CCNP CWNA
    · Share on FacebookShare on Twitter
  • APAAPA Member Posts: 959
    output from the following

    1811
    sh int fa 1

    Metro Switch
    sh int xxx

    Also why are you adjusting mss for??? I don't see any PPP connections or a need to adjust??? (Mind you I can only assume from what you have pasted)

    To me it sounds like the TCP window sizing is being cut down....hence the fast start and gradual decrease....

    As a test can you remove any mss modifications and try the download...
    CCNA | CCNA:Security | CCNP | CCIP
    JNCIA:JUNOS | JNCIA:EX | JNCIS:ENT | JNCIS:SEC
    JNCIS:SP | JNCIP:SP
    · Share on FacebookShare on Twitter
  • joshgibson82joshgibson82 Member Posts: 80 ■■□□□□□□□□
    I have completely removed all the adjust MSS statements and no change, so I put them back. I do not have access to the Metro E switch as it belongs to the ISP.
    Josh, CCNP CWNA
    · Share on FacebookShare on Twitter
  • joshgibson82joshgibson82 Member Posts: 80 ■■□□□□□□□□
    I am getting this:




    *Nov 19 11:02:11.402: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948
    *Nov 19 11:02:41.418: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948

    Also, I have removed all the adjust mss statements. all default now.
    Josh, CCNP CWNA
    · Share on FacebookShare on Twitter
  • joshgibson82joshgibson82 Member Posts: 80 ■■□□□□□□□□
    APA wrote: »
    output from the following

    1811
    sh int fa 1

    .


    #sh int f1
    FastEthernet1 is up, line protocol is up
    Hardware is PQ3_TSEC, address is 0016.47e9.0dfd (bia 0016.47e9.0dfd)
    Description: Internet Connection
    Internet address is x.x.x.x/26
    MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
    reliability 255/255, txload 1/255, rxload 1/255
    Encapsulation ARPA, loopback not set
    Keepalive set (10 sec)
    Full-duplex, 100Mb/s, 100BaseTX/FX
    ARP type: ARPA, ARP Timeout 04:00:00
    Last input 00:00:00, output 00:00:00, output hang never
    Last clearing of "show interface" counters never
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 37
    Queueing strategy: fifo
    Output queue: 0/40 (size/max)
    5 minute input rate 216000 bits/sec, 28 packets/sec
    5 minute output rate 72000 bits/sec, 22 packets/sec
    1468746 packets input, 900256088 bytes
    Received 43018 broadcasts, 0 runts, 0 giants, 0 throttles
    1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
    0 watchdog
    0 input packets with dribble condition detected
    1588942 packets output, 711440820 bytes, 0 underruns
    0 output errors, 0 collisions, 3 interface resets
    0 babbles, 0 late collision, 0 deferred
    0 lost carrier, 0 no carrier
    0 output buffer failures, 0 output buffers swapped out
    Josh, CCNP CWNA
    · Share on FacebookShare on Twitter
  • jovan88jovan88 Member Posts: 393
    which IOS are you using
    · Share on FacebookShare on Twitter
  • peanutnogginpeanutnoggin Member Posts: 1,096 ■■■□□□□□□□
    Is this happening on all clients? try adjusting your MTU size on your client... (unless its all clients having this problem).

    HTH.
    We cannot have a superior democracy with an inferior education system!

    -Mayor Cory Booker
    · Share on FacebookShare on Twitter
  • APAAPA Member Posts: 959
    sh ip inspect session (first start a legit flow from a client on the 192.168.x.x network)

    There are numerous factors that could be causing the issues you are experiencing... so we would have to go through them one by one...

    What clients are you testing from? Windows? Linux?
    CCNA | CCNA:Security | CCNP | CCIP
    JNCIA:JUNOS | JNCIA:EX | JNCIS:ENT | JNCIS:SEC
    JNCIS:SP | JNCIP:SP
    · Share on FacebookShare on Twitter
  • joshgibson82joshgibson82 Member Posts: 80 ■■□□□□□□□□
    Yes, this is happening on all clients including windows xp pro, vista, and Opensuse 10.3. I will have to wait until I get off work to do any show commands, but I know the IOS is in the 12.4 train. I will get the exact one later. I was thinking the same thing (could be IOS bug).
    Josh, CCNP CWNA
    · Share on FacebookShare on Twitter
  • SysAdmin4066SysAdmin4066 Member Posts: 443
    Have you tried removing your ip inspect statements? Just to see if it is in fact the ip inspect. You said that when you connect directly to the metro E, your download speeds are unaffected. Try by first removing all of the IP Inspect statements. Then if the problem goes away, try to add each one individually back and test each statement. What I would do is take the router back to completely open, no config besides whats absolutely necessary for routing. Then add the security one by one and test.
    In Progress: CCIE R&S Written Scheduled July 17th (Tentative)

    Next Up: CCIE R&S Lab
    · Share on FacebookShare on Twitter
  • networker050184networker050184 Mod Posts: 11,962 Mod
    SysAdmin4066 wrote: »
    Have you tried removing your ip inspect statements? Just to see if it is in fact the ip inspect. You said that when you connect directly to the metro E, your download speeds are unaffected. Try by first removing all of the IP Inspect statements. Then if the problem goes away, try to add each one individually back and test each statement. What I would do is take the router back to completely open, no config besides whats absolutely necessary for routing. Then add the security one by one and test.


    That is what I would try also. You have to narrow down the issue as much as possible.
    An expert is a man who has made all the mistakes which can be made.
    · Share on FacebookShare on Twitter
  • joshgibson82joshgibson82 Member Posts: 80 ■■□□□□□□□□
    c181x-adventerprisek9-mz.124-6.T2.bin is the IOS i'm running and also, when I removed the firewall from the interface, and the inbound ACL, the download from wireshark went extremely fast. Process of elimination now. Thanks for the idea guys!
    Josh, CCNP CWNA
    · Share on FacebookShare on Twitter
  • joshgibson82joshgibson82 Member Posts: 80 ■■□□□□□□□□
    UPgraded code to 12-4-15.T11 and configured this inspect set:


    ip inspect log drop-pkt
    ip inspect name FW_OUT http
    ip inspect name FW_OUT https
    ip inspect name FW_OUT ssh
    ip inspect name FW_OUT ftp
    ip inspect name FW_OUT echo
    ip inspect name FW_OUT tcp
    ip inspect name FW_OUT udp
    ip inspect name FW_OUT icmp
    ip inspect name FW_OUT smtp
    ip inspect name FW_OUT dns

    And got this output when downloading wireshark:
    *Nov 20 00:14:29.622: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:193142182 1500 bytes is out-of-order; expected seq:193111774. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:29.622: %FW-6-DROP_PKT: Dropping http session 69.89.22.118:80 192.168.10.10:34029 due to Out-Of-Order Segment with ip ident 40914 tcpflags 0x8010 seq.no 193142182 ack 830037319
    *Nov 20 00:14:29.666: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:14:35.286: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:194323750 1500 bytes is out-of-order; expected seq:194297686. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:40.530: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:195725414 1500 bytes is out-of-order; expected seq:195689214. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:41.666: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:14:41.686: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:195768854 1500 bytes is out-of-order; expected seq:195693558. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:42.834: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:195777542 1500 bytes is out-of-order; expected seq:195697902. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:48.118: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:196847614 1500 bytes is out-of-order; expected seq:196809966. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:49.298: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:196885262 1500 bytes is out-of-order; expected seq:196814310. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:55.854: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:198145022 1500 bytes is out-of-order; expected seq:198108822. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:14:57.694: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:15:02.663: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:199988326 1500 bytes is out-of-order; expected seq:199963710. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:02.667: %FW-6-DROP_PKT: Dropping http session 69.89.22.118:80 192.168.10.10:34029 due to Out-Of-Order Segment with ip ident 45829 tcpflags 0x8010 seq.no 199988326 ack 830037319
    *Nov 20 00:15:04.607: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:200686262 1500 bytes is out-of-order; expected seq:200645718. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:05.771: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:200719566 1500 bytes is out-of-order; expected seq:200650062. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:07.695: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:15:12.059: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:201985118 1500 bytes is out-of-order; expected seq:201940230. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:13.435: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:202583142 1500 bytes is out-of-order; expected seq:202549838. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:18.859: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TCP Timer", ipl= 4, pid= 111, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:15:18.907: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:204000734 1500 bytes is out-of-order; expected seq:203958742. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:25.539: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:205941054 1500 bytes is out-of-order; expected seq:205907750. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:30.931: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:207209502 1500 bytes is out-of-order; expected seq:207183438. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:32.695: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
    *Nov 20 00:15:36.667: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:208628542 1500 bytes is out-of-order; expected seq:208593790. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:36.667: %FW-6-DROP_PKT: Dropping http session 69.89.22.118:80 192.168.10.10:34029 due to Out-Of-Order Segment with ip ident 52060 tcpflags 0x8010 seq.no 208628542 ack 830037319
    *Nov 20 00:15:38.383: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:209287382 1500 bytes is out-of-order; expected seq:209248286. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:40.391: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:210202518 1500 bytes is out-of-order; expected seq:210176454. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:45.963: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:211649070 1500 bytes is out-of-order; expected seq:211609974. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
    *Nov 20 00:15:47.695: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8

    This version of code definitely provides more detail but I don't really know what it means.
    Josh, CCNP CWNA
    · Share on FacebookShare on Twitter
  • joshgibson82joshgibson82 Member Posts: 80 ■■□□□□□□□□
    *Nov 20 00:29:30.905: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:4261185239 1500 bytes is out-of-order; expected seq:4261149039. Reason: TCP reassembly queue overflow - session 192.168.10.10:45419 to 69.89.22.118:80



    And these!
    Josh, CCNP CWNA
    · Share on FacebookShare on Twitter
  • joshgibson82joshgibson82 Member Posts: 80 ■■□□□□□□□□
    So for anyone who cares out there..... c181x-adventerprisek9-mz.124-24.T2.bin code seems to have fixed all issues. Time will tell if the fix is permanent.
    Josh, CCNP CWNA
    · Share on FacebookShare on Twitter
  • hypnotoadhypnotoad Banned Posts: 915
    joshgibson82 wrote: »
    I am getting this:




    *Nov 19 11:02:11.402: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948
    *Nov 19 11:02:41.418: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948

    Also, I have removed all the adjust mss statements. all default now.

    I get this all the time too -- no idea what runs on 34948 but it comes in 24/7 from all over.
    · Share on FacebookShare on Twitter
  • tearofstearofs Member Posts: 112
    Try

    no ip inspect tcp reasembly queue length

    maybe?
    · Share on FacebookShare on Twitter
  • cyberguyprcyberguypr Mod Posts: 6,928 Mod
    Considering it's been a year and a half since the OP posted I think either he solved it or got fired.
    · Share on FacebookShare on Twitter
Sign In or Register to comment.