Serial errors - LOTS of 'em!

pitviperpitviper Member Posts: 1,376 ■■■■■■■□□□
OK, so we're 1000% sure it's a line issue (already replaced WIC, router, and internal cabling) – still going back and forth with the carrier.

Just out of curiosity – Any tips from the output below on specifically what could be wrong?

Serial0/3/0 is up, line protocol is up
Hardware is GT96K with integrated T1 CSU/DSU
MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 3/255, rxload 4/255
Encapsulation FRAME-RELAY IETF, loopback not set
Keepalive set (10 sec)
LMI enq sent 15881, LMI stat recvd 15872, LMI upd recvd 0, DTE LMI up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE segmentation inactive
FR SVC disabled, LAPF state down
Broadcast queue 0/64, broadcasts sent/dropped 34339/0, interface broadcasts 34339
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 1d20h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 128
Queueing strategy: Class-based queueing
Output queue: 0/1000/0 (size/max total/drops)
30 second input rate 27000 bits/sec, 15 packets/sec
30 second output rate 24000 bits/sec, 16 packets/sec
825106 packets input, 395331860 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 49 giants, 0 throttles
133241 input errors, 133241 CRC, 55461 frame, 17027 overrun, 0 ignored, 72080 abort
781056 packets output, 174405628 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
5293 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
34 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
CCNP:Collaboration, CCNP:R&S, CCNA:S, CCNA:V, CCNA, CCENT

Comments

  • tierstentiersten Member Posts: 4,505
    You can do loopback tests to see if you can narrow it down to specific portions. The telco will have to help do these though since it is unlikely that a local loopback test will do anything useful.
  • diswakdiswak Member Posts: 5 ■□□□□□□□□□
    Can you paste the output of the service module? That would be helpful to isolate the errors.
  • stlsmoorestlsmoore Member Posts: 515 ■■■□□□□□□□
    yea once you take a look at the service module/controller it should at least help narrow down when the errors are coming in. Surprised you went through all the trouble of replacing cabling, WIC, etc before having the carrier check it out though lol.
    My Cisco Blog Adventure: http://shawnmoorecisco.blogspot.com/

    Don't Forget to Add me on LinkedIn!
    https://www.linkedin.com/in/shawnrmoore
  • pitviperpitviper Member Posts: 1,376 ■■■■■■■□□□
    Thanks guys, I'll grab some more output once the circuit comes back up.
    stlsmoore wrote: »
    Surprised you went through all the trouble of replacing cabling, WIC, etc before having the carrier check it out though lol.

    We did - It's a resold circuit and the carrier(s) have been on site 5 times already. We only replaced the equipment as a cover our axx move since it’s such a hot topic w/management! :)
    CCNP:Collaboration, CCNP:R&S, CCNA:S, CCNA:V, CCNA, CCENT
  • diswakdiswak Member Posts: 5 ■□□□□□□□□□
    pitviper wrote: »
    Thanks guys, I'll grab some more output once the circuit comes back up.



    We did - It's a resold circuit and the carrier(s) have been on site 5 times already. We only replaced the equipment as a cover our axx move since it’s such a hot topic w/management! :)


    Do you have access to the router while the circuit is down? If so the service module will show you what alarms you are getting, it could help to tell you if it a Telco problem or your problem.
  • CherperCherper Member Posts: 140 ■■■□□□□□□□
    We get problems like this once in a while on T-1s to our rural locations. Often it comes down to a timing or framing problem with the telco. Check the controller settings if they don't match what the telco wants you get issues like the slips you are showing. We got rid our frame circuits since they were so painful to get reliable service on.
    Studying and Reading:

    Whatever strikes my fancy...
  • NuulNuul Member Posts: 158
    pitviper wrote: »
    It's a resold circuit and the carrier(s) have been on site 5 times already. We only replaced the equipment as a cover our axx move since it’s such a hot topic w/management! :)

    We have the same type of issue pretty frequently. My MPLS carrier is AT&T, but the LEC for the circuit might be Verizon (worst of the lot), Quest, etc. When I'm seeing CRCs and input errors like you are that's almost always a dieing smart jack. When the carrier loops up they test to the back side of the SJ. If you've done your due diligence and ruled out your hardware, extension from the SJ and the config - tell them to "prove through the smart jack." With AT&T that's a $300 charge if it turns out to not be their equipment. Basically they'll send someone to your site with a T-Bird and run patterns back and forth. They find something wrong 90% of the time in their network...and it's almost always a dieing card in their shelf.
  • sides14sides14 Member Posts: 113
    First thing I would do is buy a T1/T3 test set. They are very easy to use and you can get a great Fluke model for around $600. The great thing about the test set is that you can disconnect the RJ from the LEC (facing the router) and loop the WIC interface on the router. In addition, you can look at the signal coming from the LEC and provide them all of the information they need for diagnosing a T1 issue(Vpp, dBdsx, errors, etc). The cost is well work the relief from the headaches of dealing with the service provider.
  • pitviperpitviper Member Posts: 1,376 ■■■■■■■□□□
    The latest is that after several replaced pairs, the LEC (Verizon in this case) found a bad aerial splice 300 feet out from the DMARC – to be continued.

    Recent output for fun (which looks pretty bad):

    #sh int s0/3/0
    Serial0/3/0 is up, line protocol is up
    Hardware is GT96K with integrated T1 CSU/DSU
    MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
    reliability 255/255, txload 2/255, rxload 1/255
    Encapsulation FRAME-RELAY IETF, loopback not set
    Keepalive set (10 sec)
    LMI enq sent 4227, LMI stat recvd 4227, LMI upd recvd 0, DTE LMI up
    LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
    LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE segmentation inactive
    FR SVC disabled, LAPF state down
    Broadcast queue 0/64, broadcasts sent/dropped 9140/0, interface broadcasts 9140
    Last input 00:00:00, output 00:00:00, output hang never
    Last clearing of "show interface" counters 11:45:22
    Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 149
    Queueing strategy: Class-based queueing
    Output queue: 0/1000/0 (size/max total/drops)
    30 second input rate 2000 bits/sec, 5 packets/sec
    30 second output rate 15000 bits/sec, 36 packets/sec
    1653068 packets input, 1338189471 bytes, 0 no buffer
    Received 0 broadcasts, 0 runts, 36 giants, 0 throttles
    63843 input errors, 63842 CRC, 26718 frame, 10513 overrun, 0 ignored, 33291 abort
    1325754 packets output, 533947247 bytes, 0 underruns
    0 output errors, 0 collisions, 0 interface resets
    1411 unknown protocol drops
    0 output buffer failures, 0 output buffers swapped out
    8 carrier transitions
    DCD=up DSR=up DTR=up RTS=up CTS=up


    #sh controllers s0/3/0
    Interface Serial0/3/0
    Hardware is GT96K with Integrated FT1 CSU/DSU module
    idb at 0x47FC0E94, driver data structure at 0x47FC8640
    wic_info 0x47EDB114
    Physical Port 7, SCC Num 7
    MPSC Registers:
    MMCR_L=0x000304C0, MMCR_H=0x00000000, MPCR=0x00000100
    CHR1=0x00FE007E, CHR2=0x00000000, CHR3=0x0000064A, CHR4=0x00000000
    CHR5=0x00000000, CHR6=0x00000000, CHR7=0x00000000, CHR8=0x00000000
    CHR9=0x00000000, CHR10=0x00003008
    SDMA Registers:
    SDC=0x00002201, SDCM=0x00000080
    CRDP=0x0F5F9B30, CTDP=0x0F5FA0C0, FTDB=0x0F5FA0C0
    Main Routing Register=0x07777777 BRG Conf Register=0x00480000
    Rx Clk Routing Register=0x80000000 Tx Clk Routing Register=0x90000000
    GPP Registers:
    Conf=0x50055600, Io=0x50055600, Data=0xFBFFFFFF, Level=0x18000000
    TDM FPGA Registers:
    vmcr[0] = 0x00000005, vmcr[1] = 0x00000005,
    vmcr[2] = 0x00000000, vmcr[3] = 0x00010040
    ntrcr0 = 0x00000000, ntrcr1 = 0x00000000
    tdmcr = 0x0000006A, labcr = 0x00000000, tpllr_cr = 0x00000000
    nhr = 0x60662626, isr = 0x0000FFFF, imr = 0x00000000
    33291 input aborts on receiving flag sequence
    0 throttles, 0 enables
    10513 overruns
    0 transmitter underruns
    0 transmitter CTS losts
    1660241 rxintr, 1305033 txintr, 0 rxerr, 0 txerr
    2128137 mpsc_rx, 0 mpsc_rxerr, 96 mpsc_rlsc, 33848 mpsc_rhnt, 2120261 mpsc_rfsc
    15 mpsc_rcsc, 0 mpsc_rovr, 0 mpsc_rcdl, 0 mpsc_rckg, 0 mpsc_bper
    0 mpsc_txerr, 727810 mpsc_teidl, 0 mpsc_tudr, 0 mpsc_tctsl, 0 mpsc_tckg
    0 sdma_rx_sf, 36 sdma_rx_mfl, 10513 sdma_rx_or, 33291 sdma_rx_abr, 26718 sdma_rx_no
    0 sdma_rx_de, 0 sdma_rx_cdl, 63844 sdma_rx_ce, 0 sdma_tx_rl, 0 sdma_tx_ur, 0 sdma_tx_ctsl
    0 sdma_rx_reserr, 0 sdma_tx_reserr
    0 rx_bogus_pkts, rx_bogus_flag FALSE
    0 sdma_tx_ur_processed

    #sh service-module serial 0/3/0
    Interface Serial0/3/0
    Module type is T1/fractional
    Hardware revision is 1.2, Software revision is 20090901,
    Image checksum is 0x44F2D8, Protocol revision is 0.1
    Receiver has no alarms.
    Framing is ESF, Line Code is B8ZS, Current clock source is line,
    Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
    Last module self-test (done at startup): Passed
    Last clearing of alarm counters 11:44:58
    loss of signal : 0,
    loss of frame : 2, last occurred 00:25:29
    AIS alarm : 2, last occurred 00:25:29
    Remote alarm : 2, last occurred 00:25:22
    Module access errors : 0,
    Total Data (last 46 15 minute intervals):
    3 Line Code Violations, 2017 Path Code Violations
    4 Slip Secs, 9 Fr Loss Secs, 3 Line Err Secs, 37 Degraded Mins
    87 Errored Secs, 80 Bursty Err Secs, 9 Severely Err Secs, 0 Unavail Secs
    Data in current interval (845 seconds elapsed):
    0 Line Code Violations, 0 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
    0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
    CCNP:Collaboration, CCNP:R&S, CCNA:S, CCNA:V, CCNA, CCENT
  • pitviperpitviper Member Posts: 1,376 ■■■■■■■□□□
    sides14 wrote: »
    First thing I would do is buy a T1/T3 test set. They are very easy to use and you can get a great Fluke model for around $600.

    Interesting – Any model in particular? While it wouldn’t have helped in this case (site is remote) might be good thing to have.
    CCNP:Collaboration, CCNP:R&S, CCNA:S, CCNA:V, CCNA, CCENT
  • sides14sides14 Member Posts: 113
    The Fluke 635 QuickBERT-T1 is the model I was referencing. There are many models and manufacturers. Recommendations are always nice, but shop around (if you intend to buy) and find something you like.

    How many days total were you having issues? The great thing about having your own telecom test set (and yes the router commands are nice) is that you can loop up your interface and know 100% that your equipment is not the problem. This fast diagnostic test allows you to ride the LEC like no tomorrow (aka fast repair turnaround).
  • NuulNuul Member Posts: 158
    pitviper wrote: »
    The latest is that after several replaced pairs, the LEC (Verizon in this case) found a bad aerial splice 300 feet out from the DMARC – to be continued.

    DMARC extensions are the bane of my existence. Nearly all of my remote locations are in malls; my extensions are always getting cut by construction. You're lucky that Verizon owns your extension, I have to send a 3rd party to work on nearly all of mine. My least favorite is when AT&T says the T is clean to the SJ. I say OK so I send my wiring vendor. The wiring vendor comes out and says nope, the extension is great. So now I have to schedule a vendor meet which can only happen at 1PM for some unknown reason. Meanwhile I have a district manager hot on my ass "ZOMG WE'RE LOOSING SALES THE SKY IS FALLING!!!1!!!!" Honestly, sometimes I feel like a victim of my own success. When I was hired the mean time to repair was 5 weeks. I have it down to 24 hours or less in most cases...now they start panicking at a few hours downtime.
  • pitviperpitviper Member Posts: 1,376 ■■■■■■■□□□
    Nuul wrote: »
    DMARC extensions are the bane of my existence. Nearly all of my remote locations are in malls; my extensions are always getting cut by construction. You're lucky that Verizon owns your extension, I have to send a 3rd party to work on nearly all of mine. My least favorite is when AT&T says the T is clean to the SJ. I say OK so I send my wiring vendor. The wiring vendor comes out and says nope, the extension is great. So now I have to schedule a vendor meet which can only happen at 1PM for some unknown reason. Meanwhile I have a district manager hot on my ass "ZOMG WE'RE LOOSING SALES THE SKY IS FALLING!!!1!!!!" Honestly, sometimes I feel like a victim of my own success. When I was hired the mean time to repair was 5 weeks. I have it down to 24 hours or less in most cases...now they start panicking at a few hours downtime.

    Sorry - Other side of the DMARC (splice on the pole) our extension is OK. I know exactly what you mean though. We have several contractors who repair/redo internal wiring for us and they all seem to get pissed when we call them for an extension repair (I don’t blame them in most cases)! We’re in the same boat – Lots of official DMARCs that are several store fronts away, and most of the time, the extensions were run before the actual spaces were occupied.

    For the local sites, I typically run out with a spare router and plug it directly into the smart jack to test – And the internal wiring is usually fine.
    CCNP:Collaboration, CCNP:R&S, CCNA:S, CCNA:V, CCNA, CCENT
  • NuulNuul Member Posts: 158
    pitviper wrote: »
    Sorry - Other side of the DMARC (splice on the pole) our extension is OK.

    If that's the case I'd call BS on them testing clean like they were claiming at first. There's no way they could have ran a clean test pattern on the T with a faulty wire on the back side. On the upside you can submit an SLA for this month :D To be honest though I'm not surprised considering the LEC is Verizon. They are by far the worst LEC for this kind of thing...especially on the east coast.
  • stlsmoorestlsmoore Member Posts: 515 ■■■□□□□□□□
    pitviper wrote: »
    Thanks guys, I'll grab some more output once the circuit comes back up.



    We did - It's a resold circuit and the carrier(s) have been on site 5 times already. We only replaced the equipment as a cover our axx move since it’s such a hot topic w/management! :)

    Ahh I know that scenario all to well...best of luck man
    My Cisco Blog Adventure: http://shawnmoorecisco.blogspot.com/

    Don't Forget to Add me on LinkedIn!
    https://www.linkedin.com/in/shawnrmoore
  • diswakdiswak Member Posts: 5 ■□□□□□□□□□
    pitviper wrote: »

    #sh service-module serial 0/3/0
    Interface Serial0/3/0
    Module type is T1/fractional
    Hardware revision is 1.2, Software revision is 20090901,
    Image checksum is 0x44F2D8, Protocol revision is 0.1
    Receiver has no alarms.
    Framing is ESF, Line Code is B8ZS, Current clock source is line,
    Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
    Last module self-test (done at startup): Passed
    Last clearing of alarm counters 11:44:58
    loss of signal : 0,
    loss of frame : 2, last occurred 00:25:29
    AIS alarm : 2, last occurred 00:25:29
    Remote alarm : 2, last occurred 00:25:22
    Module access errors : 0,
    Total Data (last 46 15 minute intervals):
    3 Line Code Violations, 2017 Path Code Violations
    4 Slip Secs, 9 Fr Loss Secs, 3 Line Err Secs, 37 Degraded Mins
    87 Errored Secs, 80 Bursty Err Secs, 9 Severely Err Secs, 0 Unavail Secs
    Data in current interval (845 seconds elapsed):
    0 Line Code Violations, 0 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
    0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs



    AIS is a dead giveaway the issue is gonna be before your equipment. Unless you see LOS on your controller kick it right back to Telco telling them to recheck the circuit.
Sign In or Register to comment.