Interface Counter Issue

I had an issue today where an on-site technician called me complaining of a high ping times and loss of packets to any site remote to his location. I figured out what we going on, but I noticed something during troubleshooting that I have never seen and was wondering if anyone here as.

Started at the DC edge router that these T1s terminated to ensure both T1s were up, as Cisco 3925. Ran a "sh int | inc (des|prot|load|erro|drop)". Much to my surprise, both T1s were up, there were no errors and the rx/tx load were both sitting at 1/255. For S&Gs, I logged into the remote router, as Cisco 2901, and ran the same show command. Packets were being dropped constantly, the txLoad was at 248 on both serial interfaces and 247 on the multilink.

Thanks to Netflow I was able to figure out what was happening, but has anyone seen this issue with incorrect counters? I actually had to double check my documentation a few time to ensure I was on the right routers and looking at the correct interfaces.

Find more posts tagged with

Comments

Roguetadhg

might've been a bug in the IOS, what version is on Cisco 2901?

Monkerz

I believe the problem lies with the 3925, but both the IOS' are below:

3925:

(C3900-UNIVERSALK9-M), Version 15.0(1)M6

2901:

(C2900-UNIVERSALK9-M), Version 15.0(1)M6

kmcintosh78

Can you post the show int here?

Monkerz

I've left work for the day, but I will post when I return in the morning.

Monkerz

3925:

Serial0/2/0:0 is up, line protocol is up
  Hardware is DSX1
  Description: To_ForumIII-1of2
  MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
     reliability 255/255, [B]txload 1/255, rxload 1/255[/B]
  Encapsulation PPP, LCP Open, multilink Open
  Link is a member of Multilink bundle Multilink3, crc 16, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1d01h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair  [suspended, using FIFO]
  FIFO output queue 0/40, 0 drops
  [B]5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec[/B]
     9285198 packets input, 1324644773 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     10562373 packets output, 3407974005 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
----------------------------------------------------------------------
Serial0/2/1:0 is up, line protocol is up
  Hardware is DSX1
  Description: To_ForumIII-2of2
  MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
     reliability 255/255, [B]txload 1/255, rxload 1/255[/B]
  Encapsulation PPP, LCP Open, multilink Open
  Link is a member of Multilink bundle Multilink3, crc 16, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1d01h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: weighted fair  [suspended, using FIFO]
  FIFO output queue 0/40, 0 drops
  [B]5 minute input rate 0 bits/sec, 0 packets/sec
[/B]  [B]5 minute output rate 0 bits/sec, 0 packets/sec[/B]
     9287222 packets input, 1324762163 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     10564978 packets output, 3409076132 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
----------------------------------------------------------------------
Multilink3 is up, line protocol is up
  Hardware is multilink group interface
  Description: Bundle_to_ForumIII
  Internet address is XXX.XXX.XXX.XXX/30
  MTU 1500 bytes, BW 3072 Kbit/sec, DLY 20000 usec,
     reliability 255/255, [B]txload 1/255, rxload 1/255[/B]
  Encapsulation PPP, LCP Open, multilink Open
  Open: IPCP, CDPCP, loopback not set
  Keepalive set (10 sec)
  DTR is pulsed for 2 seconds on reset
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters 1d01h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  [B]5 minute input rate 0 bits/sec, 0 packets/sec
[/B]  [B]5 minute output rate 0 bits/sec, 0 packets/sec[/B]
     11663777 packets input, 2499361008 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     11841185 packets output, 6806172897 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

2901:

Serial0/0/0 is up, line protocol is up
  Hardware is GT96K with integrated T1 CSU/DSU
  Description: To_Corp-1of2
  MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
     reliability 255/255, [B]txload 170/255, rxload 35/255[/B]
  Encapsulation PPP, LCP Open, multilink Open
  Link is a member of Multilink bundle Multilink1, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 13w3d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 19
  Queueing strategy: weighted fair  [suspended, using FIFO]
  FIFO output queue 0/40, 19 drops
  [B]5 minute input rate 215000 bits/sec, 232 packets/sec
  5 minute output rate 1034000 bits/sec, 316 packets/sec[/B]
     374302111 packets input, 1565459620 bytes, 0 no buffer
     Received 0 broadcasts, 9589 runts, 0 giants, 0 throttles
     34486 input errors, 21140 CRC, 0 frame, 1 overrun, 0 ignored, 3756 abort
     611854481 packets output, 3391745048 bytes, 0 underruns
     0 output errors, 0 collisions, 39 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     5 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up
--------------------------------------------------------------------------
Serial0/1/0 is up, line protocol is up
  Hardware is GT96K with integrated T1 CSU/DSU
  Description: To_Corp-2of2
  MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec,
     reliability 255/255, [B]txload 170/255, rxload 35/255[/B]
  Encapsulation PPP, LCP Open, multilink Open
  Link is a member of Multilink bundle Multilink1, loopback not set
  Keepalive set (10 sec)
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 13w3d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 20
  Queueing strategy: weighted fair  [suspended, using FIFO]
  FIFO output queue 0/40, 20 drops
  [B]5 minute input rate 214000 bits/sec, 232 packets/sec
  5 minute output rate 1034000 bits/sec, 316 packets/sec[/B]
     373997072 packets input, 1436110991 bytes, 0 no buffer
     Received 0 broadcasts, 29903 runts, 0 giants, 0 throttles
     85786 input errors, 28846 CRC, 0 frame, 1 overrun, 0 ignored, 27036 abort
     611820672 packets output, 3374177339 bytes, 0 underruns
     0 output errors, 0 collisions, 38 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     7 carrier transitions
     DCD=up  DSR=up  DTR=up  RTS=up  CTS=up
----------------------------------------------------------------------------
Multilink1 is up, line protocol is up
  Hardware is multilink group interface
  Description: Bundle_To_Corp
  Internet address is XXX.XXX.XXX.XXX/30
  MTU 1500 bytes, BW 3088 Kbit/sec, DLY 20000 usec,
     reliability 255/255, [B]txload 171/255, rxload 33/255[/B]
  Encapsulation PPP, LCP Open, multilink Open
  Open: IPCP, CDPCP, loopback not set
  Keepalive set (10 sec)
  DTR is pulsed for 2 seconds on reset
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters 13w3d
  Input queue: 0/75/0/3 (size/max/drops/flushes); Total output drops: 9035056
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  [B]5 minute input rate 406000 bits/sec, 310 packets/sec
  5 minute output rate 2077000 bits/sec, 342 packets/sec[/B]
     526824652 packets input, 1286549251 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     26 input errors, 0 CRC, 14 frame, 0 overrun, 5 ignored, 7 abort
     664672125 packets output, 2420050628 bytes, 0 underruns
     0 output errors, 0 collisions, 3 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions

kmcintosh78

Check your cables.
The Telecom tech in me wants to say that one is transmitting, but hitting errors in transport.

Monkerz

But the circuit is running clean, a little over subscribed at times but clean. The errors you see on the 2901 were caused by the carrier testing.

What I don't understand is how our monitor server is polling snmp and receiving load and packets in/out from the 3925, but the routers interface counters are not registering the them.

networker050184

Sounds like a bug. Have you opened a case with Cisco?

kmcintosh78

I would second either a bug and/or possibly a glitch with the SNMP Polling Application.
Where I work, we are using an out dated and old version of Solar Winds, and it shows errors where there are none.

I would clear counters, and monitor over a 48 hour period. Don't try to fix what is not broken.

Monkerz

Oh, its broken for sure. Just have to get with Cisco to figure out why. My co-worker has opened a case, we will see what comes of it.

MAC_Addy

Very interesting - would like to know the outcome of this.