Serial errors - LOTS of 'em!

pitviper

OK, so we're 1000% sure it's a line issue (already replaced WIC, router, and internal cabling) – still going back and forth with the carrier.

Just out of curiosity – Any tips from the output below on specifically what could be wrong?

Serial0/3/0 is up, line protocol is up
Hardware is GT96K with integrated T1 CSU/DSU
MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 3/255, rxload 4/255
Encapsulation FRAME-RELAY IETF, loopback not set
Keepalive set (10 sec)
LMI enq sent 15881, LMI stat recvd 15872, LMI upd recvd 0, DTE LMI up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE segmentation inactive
FR SVC disabled, LAPF state down
Broadcast queue 0/64, broadcasts sent/dropped 34339/0, interface broadcasts 34339
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 1d20h
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 128
Queueing strategy: Class-based queueing
Output queue: 0/1000/0 (size/max total/drops)
30 second input rate 27000 bits/sec, 15 packets/sec
30 second output rate 24000 bits/sec, 16 packets/sec
825106 packets input, 395331860 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 49 giants, 0 throttles
133241 input errors, 133241 CRC, 55461 frame, 17027 overrun, 0 ignored, 72080 abort
781056 packets output, 174405628 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
5293 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
34 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up

tiersten

You can do loopback tests to see if you can narrow it down to specific portions. The telco will have to help do these though since it is unlikely that a local loopback test will do anything useful.

diswak

Can you paste the output of the service module? That would be helpful to isolate the errors.

stlsmoore

yea once you take a look at the service module/controller it should at least help narrow down when the errors are coming in. Surprised you went through all the trouble of replacing cabling, WIC, etc before having the carrier check it out though lol.

pitviper

Thanks guys, I'll grab some more output once the circuit comes back up.

stlsmoore wrote: »

Surprised you went through all the trouble of replacing cabling, WIC, etc before having the carrier check it out though lol.

We did - It's a resold circuit and the carrier(s) have been on site 5 times already. We only replaced the equipment as a cover our axx move since it’s such a hot topic w/management!

diswak

pitviper wrote: »

Thanks guys, I'll grab some more output once the circuit comes back up.



We did - It's a resold circuit and the carrier(s) have been on site 5 times already. We only replaced the equipment as a cover our axx move since it’s such a hot topic w/management!

Do you have access to the router while the circuit is down? If so the service module will show you what alarms you are getting, it could help to tell you if it a Telco problem or your problem.

Cherper

We get problems like this once in a while on T-1s to our rural locations. Often it comes down to a timing or framing problem with the telco. Check the controller settings if they don't match what the telco wants you get issues like the slips you are showing. We got rid our frame circuits since they were so painful to get reliable service on.

Nuul

pitviper wrote: »

It's a resold circuit and the carrier(s) have been on site 5 times already. We only replaced the equipment as a cover our axx move since it’s such a hot topic w/management!

We have the same type of issue pretty frequently. My MPLS carrier is AT&T, but the LEC for the circuit might be Verizon (worst of the lot), Quest, etc. When I'm seeing CRCs and input errors like you are that's almost always a dieing smart jack. When the carrier loops up they test to the back side of the SJ. If you've done your due diligence and ruled out your hardware, extension from the SJ and the config - tell them to "prove through the smart jack." With AT&T that's a $300 charge if it turns out to not be their equipment. Basically they'll send someone to your site with a T-Bird and run patterns back and forth. They find something wrong 90% of the time in their network...and it's almost always a dieing card in their shelf.

sides14

First thing I would do is buy a T1/T3 test set. They are very easy to use and you can get a great Fluke model for around $600. The great thing about the test set is that you can disconnect the RJ from the LEC (facing the router) and loop the WIC interface on the router. In addition, you can look at the signal coming from the LEC and provide them all of the information they need for diagnosing a T1 issue(Vpp, dBdsx, errors, etc). The cost is well work the relief from the headaches of dealing with the service provider.

pitviper

The latest is that after several replaced pairs, the LEC (Verizon in this case) found a bad aerial splice 300 feet out from the DMARC – to be continued.

Recent output for fun (which looks pretty bad):

#sh int s0/3/0
Serial0/3/0 is up, line protocol is up
Hardware is GT96K with integrated T1 CSU/DSU
MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation FRAME-RELAY IETF, loopback not set
Keepalive set (10 sec)
LMI enq sent 4227, LMI stat recvd 4227, LMI upd recvd 0, DTE LMI up
LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0
LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE segmentation inactive
FR SVC disabled, LAPF state down
Broadcast queue 0/64, broadcasts sent/dropped 9140/0, interface broadcasts 9140
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 11:45:22
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 149
Queueing strategy: Class-based queueing
Output queue: 0/1000/0 (size/max total/drops)
30 second input rate 2000 bits/sec, 5 packets/sec
30 second output rate 15000 bits/sec, 36 packets/sec
1653068 packets input, 1338189471 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 36 giants, 0 throttles
63843 input errors, 63842 CRC, 26718 frame, 10513 overrun, 0 ignored, 33291 abort
1325754 packets output, 533947247 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
1411 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
8 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up


#sh controllers s0/3/0
Interface Serial0/3/0
Hardware is GT96K with Integrated FT1 CSU/DSU module
idb at 0x47FC0E94, driver data structure at 0x47FC8640
wic_info 0x47EDB114
Physical Port 7, SCC Num 7
MPSC Registers:
MMCR_L=0x000304C0, MMCR_H=0x00000000, MPCR=0x00000100
CHR1=0x00FE007E, CHR2=0x00000000, CHR3=0x0000064A, CHR4=0x00000000
CHR5=0x00000000, CHR6=0x00000000, CHR7=0x00000000, CHR8=0x00000000
CHR9=0x00000000, CHR10=0x00003008
SDMA Registers:
SDC=0x00002201, SDCM=0x00000080
CRDP=0x0F5F9B30, CTDP=0x0F5FA0C0, FTDB=0x0F5FA0C0
Main Routing Register=0x07777777 BRG Conf Register=0x00480000
Rx Clk Routing Register=0x80000000 Tx Clk Routing Register=0x90000000
GPP Registers:
Conf=0x50055600, Io=0x50055600, Data=0xFBFFFFFF, Level=0x18000000
TDM FPGA Registers:
vmcr[0] = 0x00000005, vmcr[1] = 0x00000005,
vmcr[2] = 0x00000000, vmcr[3] = 0x00010040
ntrcr0 = 0x00000000, ntrcr1 = 0x00000000
tdmcr = 0x0000006A, labcr = 0x00000000, tpllr_cr = 0x00000000
nhr = 0x60662626, isr = 0x0000FFFF, imr = 0x00000000
33291 input aborts on receiving flag sequence
0 throttles, 0 enables
10513 overruns
0 transmitter underruns
0 transmitter CTS losts
1660241 rxintr, 1305033 txintr, 0 rxerr, 0 txerr
2128137 mpsc_rx, 0 mpsc_rxerr, 96 mpsc_rlsc, 33848 mpsc_rhnt, 2120261 mpsc_rfsc
15 mpsc_rcsc, 0 mpsc_rovr, 0 mpsc_rcdl, 0 mpsc_rckg, 0 mpsc_bper
0 mpsc_txerr, 727810 mpsc_teidl, 0 mpsc_tudr, 0 mpsc_tctsl, 0 mpsc_tckg
0 sdma_rx_sf, 36 sdma_rx_mfl, 10513 sdma_rx_or, 33291 sdma_rx_abr, 26718 sdma_rx_no
0 sdma_rx_de, 0 sdma_rx_cdl, 63844 sdma_rx_ce, 0 sdma_tx_rl, 0 sdma_tx_ur, 0 sdma_tx_ctsl
0 sdma_rx_reserr, 0 sdma_tx_reserr
0 rx_bogus_pkts, rx_bogus_flag FALSE
0 sdma_tx_ur_processed

#sh service-module serial 0/3/0
Interface Serial0/3/0
Module type is T1/fractional
Hardware revision is 1.2, Software revision is 20090901,
Image checksum is 0x44F2D8, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last module self-test (done at startup): Passed
Last clearing of alarm counters 11:44:58
loss of signal : 0,
loss of frame : 2, last occurred 00:25:29
AIS alarm : 2, last occurred 00:25:29
Remote alarm : 2, last occurred 00:25:22
Module access errors : 0,
Total Data (last 46 15 minute intervals):
3 Line Code Violations, 2017 Path Code Violations
4 Slip Secs, 9 Fr Loss Secs, 3 Line Err Secs, 37 Degraded Mins
87 Errored Secs, 80 Bursty Err Secs, 9 Severely Err Secs, 0 Unavail Secs
Data in current interval (845 seconds elapsed):
0 Line Code Violations, 0 Path Code Violations
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

pitviper

sides14 wrote: »

First thing I would do is buy a T1/T3 test set. They are very easy to use and you can get a great Fluke model for around $600.

Interesting – Any model in particular? While it wouldn’t have helped in this case (site is remote) might be good thing to have.

sides14

The Fluke 635 QuickBERT-T1 is the model I was referencing. There are many models and manufacturers. Recommendations are always nice, but shop around (if you intend to buy) and find something you like.

How many days total were you having issues? The great thing about having your own telecom test set (and yes the router commands are nice) is that you can loop up your interface and know 100% that your equipment is not the problem. This fast diagnostic test allows you to ride the LEC like no tomorrow (aka fast repair turnaround).

Nuul

pitviper wrote: »

The latest is that after several replaced pairs, the LEC (Verizon in this case) found a bad aerial splice 300 feet out from the DMARC – to be continued.

DMARC extensions are the bane of my existence. Nearly all of my remote locations are in malls; my extensions are always getting cut by construction. You're lucky that Verizon owns your extension, I have to send a 3rd party to work on nearly all of mine. My least favorite is when AT&T says the T is clean to the SJ. I say OK so I send my wiring vendor. The wiring vendor comes out and says nope, the extension is great. So now I have to schedule a vendor meet which can only happen at 1PM for some unknown reason. Meanwhile I have a district manager hot on my ass "ZOMG WE'RE LOOSING SALES THE SKY IS FALLING!!!1!!!!" Honestly, sometimes I feel like a victim of my own success. When I was hired the mean time to repair was 5 weeks. I have it down to 24 hours or less in most cases...now they start panicking at a few hours downtime.

pitviper

Nuul wrote: »

DMARC extensions are the bane of my existence. Nearly all of my remote locations are in malls; my extensions are always getting cut by construction. You're lucky that Verizon owns your extension, I have to send a 3rd party to work on nearly all of mine. My least favorite is when AT&T says the T is clean to the SJ. I say OK so I send my wiring vendor. The wiring vendor comes out and says nope, the extension is great. So now I have to schedule a vendor meet which can only happen at 1PM for some unknown reason. Meanwhile I have a district manager hot on my ass "ZOMG WE'RE LOOSING SALES THE SKY IS FALLING!!!1!!!!" Honestly, sometimes I feel like a victim of my own success. When I was hired the mean time to repair was 5 weeks. I have it down to 24 hours or less in most cases...now they start panicking at a few hours downtime.

Sorry - Other side of the DMARC (splice on the pole) our extension is OK. I know exactly what you mean though. We have several contractors who repair/redo internal wiring for us and they all seem to get pissed when we call them for an extension repair (I don’t blame them in most cases)! We’re in the same boat – Lots of official DMARCs that are several store fronts away, and most of the time, the extensions were run before the actual spaces were occupied.

For the local sites, I typically run out with a spare router and plug it directly into the smart jack to test – And the internal wiring is usually fine.

Nuul

pitviper wrote: »

Sorry - Other side of the DMARC (splice on the pole) our extension is OK.

If that's the case I'd call BS on them testing clean like they were claiming at first. There's no way they could have ran a clean test pattern on the T with a faulty wire on the back side. On the upside you can submit an SLA for this month To be honest though I'm not surprised considering the LEC is Verizon. They are by far the worst LEC for this kind of thing...especially on the east coast.

stlsmoore

pitviper wrote: »

Thanks guys, I'll grab some more output once the circuit comes back up.



We did - It's a resold circuit and the carrier(s) have been on site 5 times already. We only replaced the equipment as a cover our axx move since it’s such a hot topic w/management!

Ahh I know that scenario all to well...best of luck man

diswak

pitviper wrote: »

#sh service-module serial 0/3/0
Interface Serial0/3/0
Module type is T1/fractional
Hardware revision is 1.2, Software revision is 20090901,
Image checksum is 0x44F2D8, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last module self-test (done at startup): Passed
Last clearing of alarm counters 11:44:58
loss of signal : 0,
loss of frame : 2, last occurred 00:25:29
AIS alarm : 2, last occurred 00:25:29
Remote alarm : 2, last occurred 00:25:22
Module access errors : 0,
Total Data (last 46 15 minute intervals):
3 Line Code Violations, 2017 Path Code Violations
4 Slip Secs, 9 Fr Loss Secs, 3 Line Err Secs, 37 Degraded Mins
87 Errored Secs, 80 Bursty Err Secs, 9 Severely Err Secs, 0 Unavail Secs
Data in current interval (845 seconds elapsed):
0 Line Code Violations, 0 Path Code Violations
0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

AIS is a dead giveaway the issue is gonna be before your equipment. Unless you see LOS on your controller kick it right back to Telco telling them to recheck the circuit.