Interface Counter Issue
I had an issue today where an on-site technician called me complaining of a high ping times and loss of packets to any site remote to his location. I figured out what we going on, but I noticed something during troubleshooting that I have never seen and was wondering if anyone here as.
Started at the DC edge router that these T1s terminated to ensure both T1s were up, as Cisco 3925. Ran a "sh int | inc (des|prot|load|erro|drop)". Much to my surprise, both T1s were up, there were no errors and the rx/tx load were both sitting at 1/255. For S&Gs, I logged into the remote router, as Cisco 2901, and ran the same show command. Packets were being dropped constantly, the txLoad was at 248 on both serial interfaces and 247 on the multilink.
Thanks to Netflow I was able to figure out what was happening, but has anyone seen this issue with incorrect counters? I actually had to double check my documentation a few time to ensure I was on the right routers and looking at the correct interfaces.
Started at the DC edge router that these T1s terminated to ensure both T1s were up, as Cisco 3925. Ran a "sh int | inc (des|prot|load|erro|drop)". Much to my surprise, both T1s were up, there were no errors and the rx/tx load were both sitting at 1/255. For S&Gs, I logged into the remote router, as Cisco 2901, and ran the same show command. Packets were being dropped constantly, the txLoad was at 248 on both serial interfaces and 247 on the multilink.
Thanks to Netflow I was able to figure out what was happening, but has anyone seen this issue with incorrect counters? I actually had to double check my documentation a few time to ensure I was on the right routers and looking at the correct interfaces.
Comments
-
Roguetadhg
Member Posts: 2,489 ■■■■■■■■□□
might've been a bug in the IOS, what version is on Cisco 2901?In order to succeed, your desire for success should be greater than your fear of failure.
TE Threads: How to study for the CCENT/CCNA, Introduction to Cisco Exams -
Monkerz
Member Posts: 842
I believe the problem lies with the 3925, but both the IOS' are below:
3925:
(C3900-UNIVERSALK9-M), Version 15.0(1)M6
2901:
(C2900-UNIVERSALK9-M), Version 15.0(1)M6 -
kmcintosh78
Member Posts: 195
Can you post the show int here?What I am working on
CCNP Route (Currently) 80% done
CCNP Switch (Next Year)
CCNP TShoot (Next Year) -
Monkerz
Member Posts: 842
3925:
Serial0/2/0:0 is up, line protocol is up Hardware is DSX1 Description: To_ForumIII-1of2 MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec, reliability 255/255, [B]txload 1/255, rxload 1/255[/B] Encapsulation PPP, LCP Open, multilink Open Link is a member of Multilink bundle Multilink3, crc 16, loopback not set Keepalive set (10 sec) Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 1d01h Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: weighted fair [suspended, using FIFO] FIFO output queue 0/40, 0 drops [B]5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec[/B] 9285198 packets input, 1324644773 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10562373 packets output, 3407974005 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions ---------------------------------------------------------------------- Serial0/2/1:0 is up, line protocol is up Hardware is DSX1 Description: To_ForumIII-2of2 MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec, reliability 255/255, [B]txload 1/255, rxload 1/255[/B] Encapsulation PPP, LCP Open, multilink Open Link is a member of Multilink bundle Multilink3, crc 16, loopback not set Keepalive set (10 sec) Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 1d01h Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: weighted fair [suspended, using FIFO] FIFO output queue 0/40, 0 drops [B]5 minute input rate 0 bits/sec, 0 packets/sec [/B] [B]5 minute output rate 0 bits/sec, 0 packets/sec[/B] 9287222 packets input, 1324762163 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 10564978 packets output, 3409076132 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions ---------------------------------------------------------------------- Multilink3 is up, line protocol is up Hardware is multilink group interface Description: Bundle_to_ForumIII Internet address is XXX.XXX.XXX.XXX/30 MTU 1500 bytes, BW 3072 Kbit/sec, DLY 20000 usec, reliability 255/255, [B]txload 1/255, rxload 1/255[/B] Encapsulation PPP, LCP Open, multilink Open Open: IPCP, CDPCP, loopback not set Keepalive set (10 sec) DTR is pulsed for 2 seconds on reset Last input 00:00:00, output never, output hang never Last clearing of "show interface" counters 1d01h Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) [B]5 minute input rate 0 bits/sec, 0 packets/sec [/B] [B]5 minute output rate 0 bits/sec, 0 packets/sec[/B] 11663777 packets input, 2499361008 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 11841185 packets output, 6806172897 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions
2901:Serial0/0/0 is up, line protocol is up Hardware is GT96K with integrated T1 CSU/DSU Description: To_Corp-1of2 MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec, reliability 255/255, [B]txload 170/255, rxload 35/255[/B] Encapsulation PPP, LCP Open, multilink Open Link is a member of Multilink bundle Multilink1, loopback not set Keepalive set (10 sec) Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 13w3d Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 19 Queueing strategy: weighted fair [suspended, using FIFO] FIFO output queue 0/40, 19 drops [B]5 minute input rate 215000 bits/sec, 232 packets/sec 5 minute output rate 1034000 bits/sec, 316 packets/sec[/B] 374302111 packets input, 1565459620 bytes, 0 no buffer Received 0 broadcasts, 9589 runts, 0 giants, 0 throttles 34486 input errors, 21140 CRC, 0 frame, 1 overrun, 0 ignored, 3756 abort 611854481 packets output, 3391745048 bytes, 0 underruns 0 output errors, 0 collisions, 39 interface resets 0 unknown protocol drops 0 output buffer failures, 0 output buffers swapped out 5 carrier transitions DCD=up DSR=up DTR=up RTS=up CTS=up -------------------------------------------------------------------------- Serial0/1/0 is up, line protocol is up Hardware is GT96K with integrated T1 CSU/DSU Description: To_Corp-2of2 MTU 1500 bytes, BW 1544 Kbit/sec, DLY 20000 usec, reliability 255/255, [B]txload 170/255, rxload 35/255[/B] Encapsulation PPP, LCP Open, multilink Open Link is a member of Multilink bundle Multilink1, loopback not set Keepalive set (10 sec) Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 13w3d Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 20 Queueing strategy: weighted fair [suspended, using FIFO] FIFO output queue 0/40, 20 drops [B]5 minute input rate 214000 bits/sec, 232 packets/sec 5 minute output rate 1034000 bits/sec, 316 packets/sec[/B] 373997072 packets input, 1436110991 bytes, 0 no buffer Received 0 broadcasts, 29903 runts, 0 giants, 0 throttles 85786 input errors, 28846 CRC, 0 frame, 1 overrun, 0 ignored, 27036 abort 611820672 packets output, 3374177339 bytes, 0 underruns 0 output errors, 0 collisions, 38 interface resets 0 unknown protocol drops 0 output buffer failures, 0 output buffers swapped out 7 carrier transitions DCD=up DSR=up DTR=up RTS=up CTS=up ---------------------------------------------------------------------------- Multilink1 is up, line protocol is up Hardware is multilink group interface Description: Bundle_To_Corp Internet address is XXX.XXX.XXX.XXX/30 MTU 1500 bytes, BW 3088 Kbit/sec, DLY 20000 usec, reliability 255/255, [B]txload 171/255, rxload 33/255[/B] Encapsulation PPP, LCP Open, multilink Open Open: IPCP, CDPCP, loopback not set Keepalive set (10 sec) DTR is pulsed for 2 seconds on reset Last input 00:00:00, output never, output hang never Last clearing of "show interface" counters 13w3d Input queue: 0/75/0/3 (size/max/drops/flushes); Total output drops: 9035056 Queueing strategy: fifo Output queue: 0/40 (size/max) [B]5 minute input rate 406000 bits/sec, 310 packets/sec 5 minute output rate 2077000 bits/sec, 342 packets/sec[/B] 526824652 packets input, 1286549251 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 26 input errors, 0 CRC, 14 frame, 0 overrun, 5 ignored, 7 abort 664672125 packets output, 2420050628 bytes, 0 underruns 0 output errors, 0 collisions, 3 interface resets 0 unknown protocol drops 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions -
kmcintosh78
Member Posts: 195
Check your cables.
The Telecom tech in me wants to say that one is transmitting, but hitting errors in transport.What I am working on
CCNP Route (Currently) 80% done
CCNP Switch (Next Year)
CCNP TShoot (Next Year) -
Monkerz
Member Posts: 842
But the circuit is running clean, a little over subscribed at times but clean. The errors you see on the 2901 were caused by the carrier testing.
What I don't understand is how our monitor server is polling snmp and receiving load and packets in/out from the 3925, but the routers interface counters are not registering the them. -
networker050184
Mod Posts: 11,962 Mod
Sounds like a bug. Have you opened a case with Cisco?An expert is a man who has made all the mistakes which can be made. -
kmcintosh78
Member Posts: 195
I would second either a bug and/or possibly a glitch with the SNMP Polling Application.
Where I work, we are using an out dated and old version of Solar Winds, and it shows errors where there are none.
I would clear counters, and monitor over a 48 hour period. Don't try to fix what is not broken.What I am working on
CCNP Route (Currently) 80% done
CCNP Switch (Next Year)
CCNP TShoot (Next Year) -
Monkerz
Member Posts: 842
Oh, its broken for sure. Just have to get with Cisco to figure out why. My co-worker has opened a case, we will see what comes of it.
-
MAC_Addy
Member Posts: 1,740 ■■■■□□□□□□
Very interesting - would like to know the outcome of this.2017 Certification Goals:
CCNP R/S
