Help please

joshgibson82

I'm coming to this board because I know there is a wide variety of experience in here. I hope someone can help me solve a mystery in my network. I have an 1811 router connected to a metro ethernet connection. It has 3Mbps symmetrical bandwidth. When I get to speedtest.net and do a speed test, the download starts off at around 3 Meg, then quickly drops to maybe 13 Kbps. When I try to download files from certain websites (wireshark.org for example), the same thing happens. The download will start out around 400 KBps, then drop to around 5KBps. I have taken a network trace and do not see any problems. I can post that trace if someone would like to see it. I am using CBAC on the 1811 and suspect there may be a software bug with it. I have taken a laptop directly to the metro ethernet switch and connected, downloaded wireshark and other files very quickly. I am posting my Firewall config:

ip inspect log drop-pkt
ip inspect name FW_OUT cuseeme
ip inspect name FW_OUT dns
ip inspect name FW_OUT ftp
ip inspect name FW_OUT https
ip inspect name FW_OUT icmp
ip inspect name FW_OUT tftp
ip inspect name FW_OUT tcp
ip inspect name FW_OUT udp
ip inspect name FW_OUT pptp
ip inspect name FW_OUT http
ip inspect name FW_OUT h323
ip inspect name FW_OUT imap
ip inspect name FW_OUT pop3
ip inspect name FW_OUT rcmd
ip inspect name FW_OUT realaudio
ip inspect name FW_OUT rtsp
ip inspect name FW_OUT esmtp
ip inspect name FW_OUT sqlnet
ip inspect name FW_OUT streamworks
ip inspect name FW_OUT vdolive
ip inspect name FW_OUT telnet
ip inspect name FW_OUT ssh
ip ips notify SDEE

interface FastEthernet1
description Internet Connection
ip address x.x.x.x 255.255.255.192
ip access-group FW_IN in
no ip redirects
no ip proxy-arp
ip nat outside
ip inspect FW_OUT out
ip virtual-reassembly
ip tcp adjust-mss 1460
speed 100
full-duplex
no cdp enable
crypto map TO-JOSH

(the metro E switch is manually set to 100/full duplex and has an MTU of 1518 )

interface Vlan1
description $FW_INSIDE$
ip address 192.168.10.2 255.255.255.0
no ip redirects
no ip proxy-arp
ip nat inside
ip virtual-reassembly
ip tcp adjust-mss 1452

ip nat inside source route-map NAT_POOL interface FastEthernet1 overload

access-list 100 permit ip 192.168.10.0 0.0.0.255 any
route-map NAT_POOL permit 1
match ip address 100

ip access-list extended FW_IN
permit icmp any host <F1 ip address>
deny ip any any log



TOPOLOGY:

(192.168.10.0 /24 segment is VLAN 1) ----> 1811 interface F1 for WAN

---

> Metro Ethernet switch

---

> Internet

Again, most all internet sessions work fine, except when I try to do large downloads it seems. I have played with the tcp adjust-mss size and also the mtu on the F1 interface but nothing has made any difference. PLEASE HELP!!!!!!!!!!! It's driving me crazy!

joshgibson82

I'd be more than happy to run any debug commands someone would want to help troubleshoot this. I'm just out of ideas at this point.

APA

output from the following

1811
sh int fa 1

Metro Switch
sh int xxx

Also why are you adjusting mss for??? I don't see any PPP connections or a need to adjust??? (Mind you I can only assume from what you have pasted)

To me it sounds like the TCP window sizing is being cut down....hence the fast start and gradual decrease....

As a test can you remove any mss modifications and try the download...

joshgibson82

I have completely removed all the adjust MSS statements and no change, so I put them back. I do not have access to the Metro E switch as it belongs to the ISP.

joshgibson82

I am getting this:




*Nov 19 11:02:11.402: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948
*Nov 19 11:02:41.418: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948

Also, I have removed all the adjust mss statements. all default now.

joshgibson82

APA wrote: »

output from the following

1811
sh int fa 1

.

#sh int f1
FastEthernet1 is up, line protocol is up
Hardware is PQ3_TSEC, address is 0016.47e9.0dfd (bia 0016.47e9.0dfd)
Description: Internet Connection
Internet address is x.x.x.x/26
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 37
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 216000 bits/sec, 28 packets/sec
5 minute output rate 72000 bits/sec, 22 packets/sec
1468746 packets input, 900256088 bytes
Received 43018 broadcasts, 0 runts, 0 giants, 0 throttles
1 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
1588942 packets output, 711440820 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

jovan88

which IOS are you using

peanutnoggin

Is this happening on all clients? try adjusting your MTU size on your client... (unless its all clients having this problem).

HTH.

APA

sh ip inspect session (first start a legit flow from a client on the 192.168.x.x network)

There are numerous factors that could be causing the issues you are experiencing... so we would have to go through them one by one...

What clients are you testing from? Windows? Linux?

joshgibson82

Yes, this is happening on all clients including windows xp pro, vista, and Opensuse 10.3. I will have to wait until I get off work to do any show commands, but I know the IOS is in the 12.4 train. I will get the exact one later. I was thinking the same thing (could be IOS bug).

SysAdmin4066

Have you tried removing your ip inspect statements? Just to see if it is in fact the ip inspect. You said that when you connect directly to the metro E, your download speeds are unaffected. Try by first removing all of the IP Inspect statements. Then if the problem goes away, try to add each one individually back and test each statement. What I would do is take the router back to completely open, no config besides whats absolutely necessary for routing. Then add the security one by one and test.

networker050184

SysAdmin4066 wrote: »

Have you tried removing your ip inspect statements? Just to see if it is in fact the ip inspect. You said that when you connect directly to the metro E, your download speeds are unaffected. Try by first removing all of the IP Inspect statements. Then if the problem goes away, try to add each one individually back and test each statement. What I would do is take the router back to completely open, no config besides whats absolutely necessary for routing. Then add the security one by one and test.

That is what I would try also. You have to narrow down the issue as much as possible.

joshgibson82

c181x-adventerprisek9-mz.124-6.T2.bin is the IOS i'm running and also, when I removed the firewall from the interface, and the inbound ACL, the download from wireshark went extremely fast. Process of elimination now. Thanks for the idea guys!

joshgibson82

UPgraded code to 12-4-15.T11 and configured this inspect set:


ip inspect log drop-pkt
ip inspect name FW_OUT http
ip inspect name FW_OUT https
ip inspect name FW_OUT ssh
ip inspect name FW_OUT ftp
ip inspect name FW_OUT echo
ip inspect name FW_OUT tcp
ip inspect name FW_OUT udp
ip inspect name FW_OUT icmp
ip inspect name FW_OUT smtp
ip inspect name FW_OUT dns

And got this output when downloading wireshark:
*Nov 20 00:14:29.622: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:193142182 1500 bytes is out-of-order; expected seq:193111774. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:14:29.622: %FW-6-DROP_PKT: Dropping http session 69.89.22.118:80 192.168.10.10:34029 due to Out-Of-Order Segment with ip ident 40914 tcpflags 0x8010 seq.no 193142182 ack 830037319
*Nov 20 00:14:29.666: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
*Nov 20 00:14:35.286: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:194323750 1500 bytes is out-of-order; expected seq:194297686. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:14:40.530: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:195725414 1500 bytes is out-of-order; expected seq:195689214. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:14:41.666: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
*Nov 20 00:14:41.686: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:195768854 1500 bytes is out-of-order; expected seq:195693558. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:14:42.834: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:195777542 1500 bytes is out-of-order; expected seq:195697902. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:14:48.118: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:196847614 1500 bytes is out-of-order; expected seq:196809966. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:14:49.298: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:196885262 1500 bytes is out-of-order; expected seq:196814310. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:14:55.854: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:198145022 1500 bytes is out-of-order; expected seq:198108822. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:14:57.694: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
*Nov 20 00:15:02.663: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:199988326 1500 bytes is out-of-order; expected seq:199963710. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:02.667: %FW-6-DROP_PKT: Dropping http session 69.89.22.118:80 192.168.10.10:34029 due to Out-Of-Order Segment with ip ident 45829 tcpflags 0x8010 seq.no 199988326 ack 830037319
*Nov 20 00:15:04.607: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:200686262 1500 bytes is out-of-order; expected seq:200645718. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:05.771: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:200719566 1500 bytes is out-of-order; expected seq:200650062. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:07.695: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
*Nov 20 00:15:12.059: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:201985118 1500 bytes is out-of-order; expected seq:201940230. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:13.435: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:202583142 1500 bytes is out-of-order; expected seq:202549838. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:18.859: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TCP Timer", ipl= 4, pid= 111, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
*Nov 20 00:15:18.907: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:204000734 1500 bytes is out-of-order; expected seq:203958742. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:25.539: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:205941054 1500 bytes is out-of-order; expected seq:205907750. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:30.931: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:207209502 1500 bytes is out-of-order; expected seq:207183438. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:32.695: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8
*Nov 20 00:15:36.667: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:208628542 1500 bytes is out-of-order; expected seq:208593790. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:36.667: %FW-6-DROP_PKT: Dropping http session 69.89.22.118:80 192.168.10.10:34029 due to Out-Of-Order Segment with ip ident 52060 tcpflags 0x8010 seq.no 208628542 ack 830037319
*Nov 20 00:15:38.383: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:209287382 1500 bytes is out-of-order; expected seq:209248286. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:40.391: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:210202518 1500 bytes is out-of-order; expected seq:210176454. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:45.963: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:211649070 1500 bytes is out-of-order; expected seq:211609974. Reason: TCP reassembly queue overflow - session 192.168.10.10:34029 to 69.89.22.118:80
*Nov 20 00:15:47.695: %SYS-2-CHUNKINVALIDHDR: Invalid chunk header type 1 for chunk 849F85BC, data 849F9520 -Process= "TTY Background", ipl= 4, pid= 35, -Traceback= 0x809529E8 0x80428B68 0x8008F414 0x81D3DEA0 0x81D3BD14 0x81D3BF78 0x81D18390 0x81EF1D54 0x81EF21BC 0x80E17A1C 0x81A1BA70 0x80065E0C 0x80E18074 0x80E196C0 0x80E19E80 0x80E19ED8

This version of code definitely provides more detail but I don't really know what it means.

joshgibson82

*Nov 20 00:29:30.905: %FW-4-TCP_OoO_SEG: Dropping TCP Segment: seq:4261185239 1500 bytes is out-of-order; expected seq:4261149039. Reason: TCP reassembly queue overflow - session 192.168.10.10:45419 to 69.89.22.118:80



And these!

joshgibson82

So for anyone who cares out there..... c181x-adventerprisek9-mz.124-24.T2.bin code seems to have fixed all issues. Time will tell if the fix is permanent.

hypnotoad

joshgibson82 wrote: »

I am getting this:




*Nov 19 11:02:11.402: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948
*Nov 19 11:02:41.418: %FW-6-DROP_PKT: Dropping tcp pkt 69.4.231.52:80 => 192.168.10.10:34948

Also, I have removed all the adjust mss statements. all default now.

I get this all the time too -- no idea what runs on 34948 but it comes in 24/7 from all over.

tearofs

Try

no ip inspect tcp reasembly queue length

maybe?

cyberguypr

Considering it's been a year and a half since the OP posted I think either he solved it or got fired.