Buffer Misses

mzinz

One of my main routers is getting a bunch of buffer misses, resulting in packet drops. I have pasted output below:


Genoa-C3845-Rtr# sh buff
Buffer elements:
1117 in free list (1119 max allowed)
445357378 hits, 0 misses, 619 created

Public buffer pools:
Small buffers, 104 bytes (total 176, permanent 70, peak 230 @ 7w0d):
150 in free list (20 min, 150 max allowed)
596539015 hits, 350633 misses, 147852 trims, 147958 created
124863 failures (0 no memory)
Middle buffers, 600 bytes (total 35, permanent 35, peak 167 @ 7w0d):
31 in free list (10 min, 150 max allowed)
1286575676 hits, 32303 misses, 9766 trims, 9766 created
22400 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50, peak 95 @ 7w0d):
49 in free list (5 min, 150 max allowed)
88445460 hits, 8034 misses, 910 trims, 910 created
7430 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10, peak 16 @ 7w0d):
9 in free list (0 min, 100 max allowed)
1163206 hits, 6472 misses, 97 trims, 97 created
6471 failures (0 no memory)
Large buffers, 5024 bytes (total 1, permanent 0, peak 7 @ 7w0d):
1 in free list (0 min, 10 max allowed)
106 hits, 6365 misses, 3210 trims, 3211 created
6365 failures (0 no memory)
Huge buffers, 18024 bytes (total 1, permanent 0, peak 8 @ 7w0d):
1 in free list (0 min, 4 max allowed)
193969 hits, 6355 misses, 3081 trims, 3082 created
6268 failures (0 no memory)


This was happening months ago, but it would be just a couple small misses a day. I raised the small buffer pool as necessary, but it didn't seem to help much - maybe I wasn't raising it enough.

Anyway, I checked again today, and there are tons of misses from all pool, and a lot of drops.

Also, here's a SMTP graph showing all of the misses:
Whoops! - TinyGrab

Any thoughts? Obviously I need more in the small and middle pools... Should I be raising the min, max, and permanent? I guess I'm a little reserved because I'm afraid of reserving too much memory...

Thanks

networker050184

What is the router doing (services, traffic level, tunnels etc.)? I've never seen the defaults need changed before on any router. It would probably be a better idea to find out the root cause rather than changing buffers all around to work around the real issue. I'm assuming you have a lot more traffic being process switched than you should.

mzinz

networker050184 wrote: »

What is the router doing (services, traffic level, tunnels etc.)? I've never seen the defaults need changed before on any router. It would probably be a better idea to find out the root cause rather than changing buffers all around to work around the real issue. I'm assuming you have a lot more traffic being process switched than you should.

Good point.

Our topology will soon be changing... Right now, the router is terminating VPN's, and is acting as a router on a stick for several VLANs in our datacenter.

We have a pair of 3750's on the way to replace it, at which point it will be used to pass traffic between our datacenter and an MPLS cloud and handle VPN.

networker050184

If I had to guess I'd say its something with the VPN traffic being processed switched. You might want to look at the product sheets for your router to ensure everything you are doing on it is supported via CEF and does not require a punt.

mzinz

networker050184 wrote: »

If I had to guess I'd say its something with the VPN traffic being processed switched. You might want to look at the product sheets for your router to ensure everything you are doing on it is supported via CEF and does not require a punt.

99% of the traffic is from the VLAN routing, so I assumed it was caused by that, but anything is possible.

It is a 3845. There are about ~5 site-to-site VPN's, and between ~5-20 users doing remote access VPN at any given time.