Interface resets on an eth interface?
creamy_stew
Member Posts: 406 ■■■□□□□□□□
in CCNP
What, if anything, are Interface Resets indicative of on an ethernet interface?
Usually, I see perhaps a couple on an interface whose counters haven't been reset for a long time. Now, I'm having sporadic but major problems with an c1812. It loses OSPF connections to it's neighbors, sometimes it seems it just loses the network connection altogether.
Initially, I just looked at whether the interface had gone down, CRC errors FCS etc, but no problem there. Later, I noticed that the "interface resets" counter in sh int increases at a rediculous rate. What info I can find online only refers to serial interfaces and I don't really understand what they mean anyway. One comment seemed to indicate that runts are somehow related to them, but I get no runts at at all.
sh int output for your enjoyment:
FastEthernet0 is up, line protocol is up
Hardware is PQ3_TSEC, address is 001f.9e8c.xxxx (bia 001f.9e8c.xxxx)
Internet address is 10.x.y.z/25
MTU 1500 bytes, BW 10000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3000 bits/sec, 3 packets/sec
5 minute output rate 2000 bits/sec, 2 packets/sec
1011966 packets input, 367334058 bytes
Received 105294 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
983881 packets output, 76858816 bytes, 0 underruns
0 output errors, 0 collisions, 9453 interface resets <---- Holy cow!
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Usually, I see perhaps a couple on an interface whose counters haven't been reset for a long time. Now, I'm having sporadic but major problems with an c1812. It loses OSPF connections to it's neighbors, sometimes it seems it just loses the network connection altogether.
Initially, I just looked at whether the interface had gone down, CRC errors FCS etc, but no problem there. Later, I noticed that the "interface resets" counter in sh int increases at a rediculous rate. What info I can find online only refers to serial interfaces and I don't really understand what they mean anyway. One comment seemed to indicate that runts are somehow related to them, but I get no runts at at all.
sh int output for your enjoyment:
FastEthernet0 is up, line protocol is up
Hardware is PQ3_TSEC, address is 001f.9e8c.xxxx (bia 001f.9e8c.xxxx)
Internet address is 10.x.y.z/25
MTU 1500 bytes, BW 10000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3000 bits/sec, 3 packets/sec
5 minute output rate 2000 bits/sec, 2 packets/sec
1011966 packets input, 367334058 bytes
Received 105294 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
983881 packets output, 76858816 bytes, 0 underruns
0 output errors, 0 collisions, 9453 interface resets <---- Holy cow!
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Comments
-
networker050184
Mod Posts: 11,962 Mod
What is it connected to and whats the config look like?An expert is a man who has made all the mistakes which can be made. -
chmorin
Member Posts: 1,446 ■■■■■□□□□□
Forsaken_GA wrote: »check the duplex on the other end. And try a different cable.
Duplex issues would usually be throwing output errors I think... But I would still confirm that both sides are agreeing on the speed and duplex configurations.Currently PursuingWGU (BS in IT Network Administration) - 52%| CCIE:Voice Written - 0% (0/200 Hours)mikej412 wrote:Cisco Networking isn't just a job, it's a Lifestyle. -
Forsaken_GA
Member Posts: 4,024
Duplex issues would usually be throwing output errors I think... But I would still confirm that both sides are agreeing on the speed and duplex configurations.
It would, but I've seen noisy servers with duplex mismatches cause interface resets before (not at the rate shown in his output though).
duplex, along with dns, is one of the things I always check when things start breaking, better to rule it out early than find out it's the cause a couple hours down the road.
More likely than not, he's got a bad cable or transceiver -
creamy_stew
Member Posts: 406 ■■■□□□□□□□
networker050184 wrote: »What is it connected to and whats the config look like?
In a broader sense, it connects to a leased point-to-multipoint VLAN. The handoff switch is usually a cisco or extreme networks, though I haven't seen it with my own eyes. Also, the plot just thickened: Between the provider switch and this router, there's single-mode fiber with a media converter at each end. No-one seems to know who actually owns the converters
Anyway: sh run brie (hope I removed all sensitive info)
Current configuration : 10293 bytes
!
! No configuration change since last restart
!
version 12.4
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime
service timestamps log datetime localtime
service password-encryption
service sequence-numbers
!
hostname somerouter
!
boot-start-marker
boot-end-marker
!
logging buffered 51200 informational
!
aaa new-model
!
!
aaa group server radius RadiusServers
server 10.22.0.10 auth-port 1812 acct-port 1813
server 10.22.0.11 auth-port 1812 acct-port 1813
!
aaa authentication login default group RadiusServers local
aaa authorization exec default group RadiusServers local
!
!
aaa session-id common
clock timezone Berlin 1
clock summer-time Berlin recurring last Sun Mar 2:00 last Sun Oct 3:00
!
<snip crypto>
!
!
ip cef
no ip dhcp use vrf connected
ip dhcp excluded-address 10.22.7.1 10.22.7.19
ip dhcp excluded-address 10.22.27.1 10.22.27.19
!
ip dhcp pool dhcp-pool
network 10.22.7.0 255.255.255.0
domain-name somedomain.se
dns-server 10.22.0.10 10.22.0.11
netbios-name-server 10.22.0.10 10.22.0.11
netbios-node-type p-node
default-router 10.22.7.1
lease 0 2
!
ip dhcp pool voip-pool
network 10.22.27.0 255.255.255.0
domain-name somedomain.se
dns-server 10.22.0.10 10.22.0.11
default-router 10.22.27.1
!
!
no ip domain lookup
ip domain name somedomain.se
ip auth-proxy max-nodata-conns 3
ip admission max-nodata-conns 3
!
multilink bundle-name authenticated
!
!
!
spanning-tree uplinkfast
spanning-tree backbonefast
vtp domain somedomain
vtp mode transparent
<snip users>
!
!
archive
log config
hidekeys
!
!
vlan 20
name VOIP
!
!
<snip policy/classmaps>
!
!
!
!
interface Loopback0
ip address 10.22.16.136 255.255.255.255
!
interface FastEthernet0
ip address 10.22.16.9 255.255.255.128
speed auto
full-duplex
service-policy output SHAPER
!
interface FastEthernet1
no ip address
shutdown
duplex auto
speed auto
!
interface BRI0
no ip address
encapsulation hdlc
shutdown
!
interface FastEthernet2
switchport trunk allowed vlan 1,20,1002-1005
switchport mode trunk
!
interface FastEthernet3
switchport trunk allowed vlan 1,20,1002-1005
switchport mode trunk
spanning-tree portfast
!
interface FastEthernet4
switchport trunk allowed vlan 1,20,1002-1005
switchport mode trunk
spanning-tree portfast
!
interface FastEthernet5
switchport trunk allowed vlan 1,20,1002-1005
switchport mode trunk
spanning-tree portfast
!
interface FastEthernet6
switchport trunk allowed vlan 1,20,1002-1005
switchport mode trunk
spanning-tree portfast
!
interface FastEthernet7
switchport trunk allowed vlan 1,20,1002-1005
switchport mode trunk
spanning-tree portfast
!
interface FastEthernet8
switchport trunk allowed vlan 1,20,1002-1005
switchport mode trunk
spanning-tree portfast
!
interface FastEthernet9
switchport trunk allowed vlan 1,20,1002-1005
switchport mode trunk
spanning-tree portfast
!
interface Vlan1
description $ETH-SW-LAUNCH$$INTF-INFO-FE 2$
ip address 10.22.7.1 255.255.255.0
ip tcp adjust-mss 1452
!
interface Vlan20
description VOIP
ip address 10.22.27.1 255.255.255.0
ip access-group VOIP-traffic in
no ip redirects
no ip unreachables
no ip proxy-arp
ip route-cache flow
ip tcp adjust-mss 1452
no autostate
!
router ospf 1
log-adjacency-changes
passive-interface FastEthernet1
passive-interface Vlan1
passive-interface Vlan20
network 10.22.7.0 0.0.0.255 area 0
network 10.22.16.0 0.0.0.127 area 0
network 10.22.16.136 0.0.0.0 area 0
network 10.22.27.0 0.0.0.255 area 0
!
ip forward-protocol nd
!
!
ip http server
ip http access-class 23
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
!
<snip acls>
snmp-server community <snip>
snmp-server ifindex persist
snmp-server trap-source Loopback0
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps vrrp
snmp-server enable traps tty
snmp-server enable traps eigrp
snmp-server enable traps flash insertion removal
snmp-server enable traps isdn call-information
snmp-server enable traps isdn layer2
snmp-server enable traps isdn chan-not-avail
snmp-server enable traps isdn ietf
snmp-server enable traps envmon
snmp-server enable traps disassociate
snmp-server enable traps deauthenticate
snmp-server enable traps authenticate-fail
snmp-server enable traps dot11-qos
snmp-server enable traps switch-over
snmp-server enable traps rogue-ap
snmp-server enable traps wlan-wep
snmp-server enable traps aaa_server
snmp-server enable traps atm subif
snmp-server enable traps bgp
snmp-server enable traps bulkstat collection transfer
snmp-server enable traps cnpd
snmp-server enable traps config-copy
snmp-server enable traps config
snmp-server enable traps entity
snmp-server enable traps resource-policy
snmp-server enable traps event-manager
snmp-server enable traps frame-relay
snmp-server enable traps frame-relay subif
snmp-server enable traps hsrp
snmp-server enable traps ipmulticast
snmp-server enable traps mpls ldp
snmp-server enable traps mpls traffic-eng
snmp-server enable traps mpls vpn
snmp-server enable traps msdp
snmp-server enable traps mvpn
snmp-server enable traps ospf state-change
snmp-server enable traps ospf errors
snmp-server enable traps ospf retransmit
snmp-server enable traps ospf lsa
snmp-server enable traps ospf cisco-specific state-change nssa-trans-change
snmp-server enable traps ospf cisco-specific state-change shamlink interface-old
snmp-server enable traps ospf cisco-specific state-change shamlink neighbor
snmp-server enable traps ospf cisco-specific errors
snmp-server enable traps ospf cisco-specific retransmit
snmp-server enable traps ospf cisco-specific lsa
snmp-server enable traps pim neighbor-change rp-mapping-change invalid-pim-message
snmp-server enable traps pppoe
snmp-server enable traps cpu threshold
snmp-server enable traps rsvp
snmp-server enable traps ipsla
snmp-server enable traps syslog
snmp-server enable traps l2tun session
snmp-server enable traps l2tun pseudowire status
snmp-server enable traps pw vc
snmp-server enable traps firewall serverstatus
snmp-server enable traps isakmp policy add
snmp-server enable traps isakmp policy delete
snmp-server enable traps isakmp tunnel start
snmp-server enable traps isakmp tunnel stop
snmp-server enable traps ipsec cryptomap add
snmp-server enable traps ipsec cryptomap delete
snmp-server enable traps ipsec cryptomap attach
snmp-server enable traps ipsec cryptomap detach
snmp-server enable traps ipsec tunnel start
snmp-server enable traps ipsec tunnel stop
snmp-server enable traps ipsec too-many-sas
snmp-server host <snip>
snmp-server host <snip>
no cdp run
!
!
!
!
!
radius-server host 10.22.0.10 auth-port 1812 acct-port 1813 key 7 <snip>
radius-server host 10.22.0.11 auth-port 1812 acct-port 1813 key 7 <snip>
radius-server vsa send authentication
!
control-plane
!
<snip banner>
!
line con 0
logging synchronous
line aux 0
line vty 0 4
access-class 23 in
privilege level 15
logging synchronous
transport input ssh
line vty 5 15
access-class 23 in
privilege level 15
logging synchronous
transport input ssh
!
ntp clock-period 17180089
ntp server 10.22.16.1 prefer
ntp server 10.22.16.2
end -
Forsaken_GA
Member Posts: 4,024
creamy_stew wrote: »In a broader sense, it connects to a leased point-to-multipoint VLAN. The handoff switch is usually a cisco or extreme networks, though I haven't seen it with my own eyes. Also, the plot just thickened: Between the provider switch and this router, there's single-mode fiber with a media converter at each end. No-one seems to know who actually owns the converters
I'd bet dollars to donuts the media converter is the root cause. Hate those damned things. -
networker050184
Mod Posts: 11,962 Mod
First place I'd start is the converters. You are probably losing link on the fiber but not the physical link to the converter.An expert is a man who has made all the mistakes which can be made.
-
creamy_stew
Member Posts: 406 ■■■□□□□□□□
networker050184 wrote: »First place I'd start is the converters. You are probably losing link on the fiber but not the physical link to the converter.
Oh, right! Some converters keep the link up even if the link on the other side dies. That must make for some dynamic routing fun
Could this be the cause of the interface resets? I still don't get what they actually mean -
Heero
Member Posts: 486
creamy_stew wrote: »Oh, right! Some converters keep the link up even if the link on the other side dies. That must make for some dynamic routing fun
Could this be the cause of the interface resets? I still don't get what they actually mean
I've experienced the faulty link loss detection with a CWDM setup before. It is quite annoying, but its a good place to start. -
Forsaken_GA
Member Posts: 4,024
creamy_stew wrote: »Oh, right! Some converters keep the link up even if the link on the other side dies. That must make for some dynamic routing fun
Could this be the cause of the interface resets? I still don't get what they actually mean
it basically means that the interface had packets all queued up but they didn't get transmitted, so it resets the interface -
creamy_stew
Member Posts: 406 ■■■□□□□□□□
Forsaken_GA wrote: »it basically means that the interface had packets all queued up but they didn't get transmitted, so it resets the interface
So how does that happen? If the interface is up, why doesn't it just spew out those eth frames? I think I'm about to get one of those elusive "Aha" moments -
Forsaken_GA
Member Posts: 4,024
creamy_stew wrote: »So how does that happen? If the interface is up, why doesn't it just spew out those eth frames? I think I'm about to get one of those elusive "Aha" moments
I'm not entirely sure when it comes to ethernet. I know for serial interfaces, it's because the interface has it's doubts about the line protocol actually being up, so I'd imagine it to be something similar. -
wolverene13
Member Posts: 87 ■■□□□□□□□□
creamy_stew wrote: »Oh, right! Some converters keep the link up even if the link on the other side dies. That must make for some dynamic routing fun
Could this be the cause of the interface resets? I still don't get what they actually mean
Interface resets indicate that the port bounced. Basically you have a bouncing circuit. Media converters are circuit equipment and owned by the provider. I work for a large WAN provider and we have problems with those stupid media converters all the time (usually Omnitrons). I would first open a trouble ticket with them and then get real pushy and request to be moved to straight fiber.Currently Studying: CCIP - 642-611 - MPLS
Occupation: Tier II NOC Tech - Centurylink
CCIP Progress: [x] BSCI
[x] BGP
[ ] MPLS
[ ] QoS -
creamy_stew
Member Posts: 406 ■■■□□□□□□□
wolverene13 wrote: »Interface resets indicate that the port bounced. Basically you have a bouncing circuit. Media converters are circuit equipment and owned by the provider. I work for a large WAN provider and we have problems with those stupid media converters all the time (usually Omnitrons). I would first open a trouble ticket with them and then get real pushy and request to be moved to straight fiber.
Hm, I thought a bounce on an ethernet interface was were the port loses the link? This isn't happening here. There are around double digits worth of interface resets per second. However, I see no Link up/down/flaps. The provider didn't see any up/downs either.
Do you mean the fiber interfaces on the converters are flapping, and those flaps, in turn, cause the resets on my router? -
Forsaken_GA
Member Posts: 4,024
creamy_stew wrote: »Hm, I thought a bounce on an ethernet interface was were the port loses the link? This isn't happening here. There are around double digits worth of interface resets per second. However, I see no Link up/down/flaps. The provider didn't see any up/downs either.
Do you mean the fiber interfaces on the converters are flapping, and those flaps, in turn, cause the resets on my router?
If the fiber converter is munged and isn't, you know, converting, both sides of the physical link can still appear to be up, so you wouldn't see any link up/down messages. -
wolverene13
Member Posts: 87 ■■□□□□□□□□
creamy_stew wrote: »Hm, I thought a bounce on an ethernet interface was were the port loses the link? This isn't happening here. There are around double digits worth of interface resets per second. However, I see no Link up/down/flaps. The provider didn't see any up/downs either.
Do you mean the fiber interfaces on the converters are flapping, and those flaps, in turn, cause the resets on my router?
Unless your provider has link propagation turned on in the media converter, neither of you will see a bounce if it happens on the fiber itself. Both you and the carrier will only be able to see events that occur between the media converter and your respective devices. When your router sees the link as being "up," it is only seeing it's connection as up to the media converter and is not actually seeing the link as being up to the provider's device on the other end of the circuit. Your provider will see the same thing on their end. I would have your carrier run head to head testing on the circuit to either prove or refute this assumption. This is going to require a tech in the CO and a tech at your site, both with test sets. They will then push traffic to each others' test sets and be able to determine if there is an issue. However, normally if they don't find an issue after doing this, you'll get billed for the dispatch. You won't get a bill if they actually find a problem on their end (or at least that's how it works with my company). So, I would thoroughly check your end before requesting that they do this. Another thing; are you actually seeing a problem as a result of this, or are you just noticing the interface resets and want to know why this is showing up? If no problem is arising as a result of the resets, it may just be a cosmetic bug in the version of IOS you are running.Currently Studying: CCIP - 642-611 - MPLS
Occupation: Tier II NOC Tech - Centurylink
CCIP Progress: [x] BSCI
[x] BGP
[ ] MPLS
[ ] QoS -
creamy_stew
Member Posts: 406 ■■■□□□□□□□
wolverene13 wrote: »Unless your provider has link propagation turned on in the media converter, neither of you will see a bounce if it happens on the fiber itself. Both you and the carrier will only be able to see events that occur between the media converter and your respective devices. When your router sees the link as being "up," it is only seeing it's connection as up to the media converter and is not actually seeing the link as being up to the provider's device on the other end of the circuit. Your provider will see the same thing on their end. I would have your carrier run head to head testing on the circuit to either prove or refute this assumption. This is going to require a tech in the CO and a tech at your site, both with test sets. They will then push traffic to each others' test sets and be able to determine if there is an issue. However, normally if they don't find an issue after doing this, you'll get billed for the dispatch. You won't get a bill if they actually find a problem on their end (or at least that's how it works with my company). So, I would thoroughly check your end before requesting that they do this. Another thing; are you actually seeing a problem as a result of this, or are you just noticing the interface resets and want to know why this is showing up? If no problem is arising as a result of the resets, it may just be a cosmetic bug in the version of IOS you are running.
About seeing problems, here's an OSPF log from the designated router from when the kaka hit the fan (10.22.16.136 is the router I was talking about):
edit: Curiously, I didn't get any alarms from 10.22.16.136 until approx. 12
x. This seems to suggest, to me, that they had a very low bandwidth loop that was picking up speed. and OSPF was able to converge fast enough for the polling not to notice before that time?
edit2: I really should learn how to make those neat in-post scrolling frames
I just never posted logs/configs until now.
DESIGNATED-ROUTER01#sh logg | i OSPF
097566: Sep 29 11:08:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.132 on FastEthernet0 from FULL to DOWN, Neighbor Down: Dead timer expired
097567: Sep 29 11:09:01: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.132 on FastEthernet0 from LOADING to FULL, Loading Done
097573: Sep 29 11:15:11: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.142 on FastEthernet0 from FULL to DOWN, Neighbor Down: Dead timer expired
097574: Sep 29 11:16:11: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.142 on FastEthernet0 from LOADING to FULL, Loading Done
097575: Sep 29 11:19:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097576: Sep 29 11:20:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097577: Sep 29 11:20:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097578: Sep 29 11:21:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097579: Sep 29 11:22:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097597: Sep 29 11:57:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097599: Sep 29 11:58:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097609: Sep 29 11:59:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097610: Sep 29 12:00:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097615: Sep 29 12:00:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097645: Sep 29 12:01:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097646: Sep 29 12:02:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097647: Sep 29 12:03:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097648: Sep 29 12:04:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097653: Sep 29 12:05:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097654: Sep 29 12:06:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097655: Sep 29 12:07:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097656: Sep 29 12:08:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097657: Sep 29 12:08:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097658: Sep 29 12:09:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097659: Sep 29 12:10:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097660: Sep 29 12:11:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097661: Sep 29 12:12:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097662: Sep 29 12:13:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097663: Sep 29 12:14:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097664: Sep 29 12:14:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097665: Sep 29 12:15:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097666: Sep 29 12:16:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097667: Sep 29 12:18:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097668: Sep 29 12:19:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097669: Sep 29 12:20:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097671: Sep 29 12:21:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097672: Sep 29 12:21:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097673: Sep 29 12:22:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097674: Sep 29 12:24:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097675: Sep 29 12:25:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097676: Sep 29 12:26:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097677: Sep 29 12:26:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097678: Sep 29 12:27:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097679: Sep 29 12:28:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097680: Sep 29 12:29:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097681: Sep 29 12:30:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097682: Sep 29 12:31:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097683: Sep 29 12:32:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097684: Sep 29 12:32:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097685: Sep 29 12:33:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097686: Sep 29 12:34:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097687: Sep 29 12:35:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097688: Sep 29 12:36:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097689: Sep 29 12:37:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097690: Sep 29 12:38:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097691: Sep 29 12:39:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097692: Sep 29 12:40:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097693: Sep 29 12:41:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097694: Sep 29 12:41:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097695: Sep 29 12:42:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097696: Sep 29 12:43:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097697: Sep 29 12:45:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097698: Sep 29 12:46:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097699: Sep 29 12:46:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097701: Sep 29 12:48:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097702: Sep 29 12:49:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097703: Sep 29 12:50:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097704: Sep 29 12:51:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097736: Sep 29 12:52:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097737: Sep 29 12:53:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097738: Sep 29 12:54:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097739: Sep 29 12:54:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097740: Sep 29 13:05:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097741: Sep 29 13:06:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097743: Sep 29 13:07:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097744: Sep 29 13:08:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097746: Sep 29 13:08:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097747: Sep 29 13:09:48: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097748: Sep 29 13:10:38: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097749: Sep 29 13:11:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097750: Sep 29 13:12:48: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097751: Sep 29 13:13:38: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097752: Sep 29 13:14:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097753: Sep 29 13:15:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097754: Sep 29 13:16:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097755: Sep 29 13:16:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097756: Sep 29 13:17:48: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097757: Sep 29 13:18:38: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097758: Sep 29 13:19:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097763: Sep 29 13:20:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097764: Sep 29 13:21:38: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097765: Sep 29 13:22:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097766: Sep 29 13:23:48: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097767: Sep 29 13:24:38: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097768: Sep 29 13:25:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097770: Sep 29 13:26:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097771: Sep 29 13:27:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097772: Sep 29 13:28:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097774: Sep 29 13:29:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097776: Sep 29 13:30:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097790: Sep 29 13:54:39: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from FULL to DOWN, Neighbor Down: Dead timer expired
097791: Sep 29 13:55:10: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done -
wolverene13
Member Posts: 87 ■■□□□□□□□□
creamy_stew wrote: »About seeing problems, here's an OSPF log from the designated router from when the kaka hit the fan (10.22.16.136 is the router I was talking about):
edit: Curiously, I didn't get any alarms from 10.22.16.136 until approx. 12x. This seems to suggest, to me, that they had a very low bandwidth loop that was picking up speed. and OSPF was able to converge fast enough for the polling not to notice before that time?
edit2: I really should learn how to make those neat in-post scrolling frames
I just never posted logs/configs until now.
DESIGNATED-ROUTER01#sh logg | i OSPF
097566: Sep 29 11:08:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.132 on FastEthernet0 from FULL to DOWN, Neighbor Down: Dead timer expired
097567: Sep 29 11:09:01: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.132 on FastEthernet0 from LOADING to FULL, Loading Done
097573: Sep 29 11:15:11: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.142 on FastEthernet0 from FULL to DOWN, Neighbor Down: Dead timer expired
097574: Sep 29 11:16:11: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.142 on FastEthernet0 from LOADING to FULL, Loading Done
097575: Sep 29 11:19:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097576: Sep 29 11:20:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
097577: Sep 29 11:20:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
I noticed this is the output from the log with everything that has the word "OSPF" in it. Are there any other pertinent events occurring too (like other logs without "OSPF" in them)? Another thing that I noticed is that OSPF is only going from LOADING to FULL most of the time and not really going down. This indicates a logical issue if that's the case (it still may be resulting from a physical problem, but it's logical in nature because all that is happening is that the two routers are updating their routing tables based on the information they have provided one another). Another thing is that the log entries are exactly 50 seconds apart (with the exception of the ones in the beginning). STP takes 50 seconds to converge, so a Layer 2 loop is entirely possible. Do you happen to know what type of network your provider has you on? Is it Metro Ethernet, ATM, MPLS, or Frame?Currently Studying: CCIP - 642-611 - MPLS
Occupation: Tier II NOC Tech - Centurylink
CCIP Progress: [x] BSCI
[x] BGP
[ ] MPLS
[ ] QoS -
phoeneous
Member Posts: 2,333 ■■■■■■■□□□
wolverene13 wrote: »Basically you have a bouncing circuit.
That's what it sounds like to me. Anything good in sh log? -
creamy_stew
Member Posts: 406 ■■■□□□□□□□
wolverene13 wrote: »Another thing is that the log entries are exactly 50 seconds apart (with the exception of the ones in the beginning). STP takes 50 seconds to converge, so a Layer 2 loop is entirely possible. Do you happen to know what type of network your provider has you on? Is it Metro Ethernet, ATM, MPLS, or Frame?
Well, this (the DR) also NATs some traffic (it announces a default route) (so you get all sorts of interesting logs on the Internet-facing interface, and it also logs some acls, so the "i OSPF" was strictly a courtesy)
I also did an " i include" for both the actual interface and the loopback(neighbor) interface. That didn't provide any more info. If there's a particular time period you're intereted in, I can dig into the syslogs and give give you the whole shebang (It's just a pain to look through them to verify that there's no senistive info leaking out:).
As I said I think they had a loop, I also believe they are running eth somewhere between "us" and "them". -
creamy_stew
Member Posts: 406 ■■■□□□□□□□
That's what it sounds like to me. Anything good in sh log?
Nothing really good, AFAICT.
The outward facing interface never loses link
The OSPF process times out all or most of the other routers
On the DR, I can see it going from LOADINg to FULL a gazillion times, occasioanlly going from FULL to DOWN (from memory).
Apart from that, everything semms hunky-dory(?)
This is obviously an STP problem somwhere along the path. What I don't get is those interface resets. Do they mean I should fix something? -
wolverene13
Member Posts: 87 ■■□□□□□□□□
creamy_stew wrote: »Well, this (the DR) also NATs some traffic (it announces a default route) (so you get all sorts of interesting logs on the Internet-facing interface, and it also logs some acls, so the "i OSPF" was strictly a courtesy)
Internet-facing interface? If that interface is the same one that you are seeing the interface resets on, is this OSPF relationship over a GRE tunnel or something? If that's the case, there are about 1,000 things that could be happening. Did you do any traceroutes when this was going on (or is it still happening)?creamy_stew wrote: »I also did an " i include" for both the actual interface and the loopback(neighbor) interface. That didn't provide any more info. If there's a particular time period you're intereted in, I can dig into the syslogs and give give you the whole shebang (It's just a pain to look through them to verify that there's no senistive info leaking out:).
I think a snapshot of the logs starting with the entry that is right after one that references OSPF going from LOADING to FULL all the way until just before the next OSPF LOADING/FULL entry would be helpful. I think seeing the logs from right after the LOADING/FULL entry to the FULL to DOWN would also show something useful. Obviously you should omit anything you don't want others to see.creamy_stew wrote: »As I said I think they had a loop, I also believe they are running eth somewhere between "us" and "them".
If they are doing QinQ tunneling at all, I would almost bet my next paycheck this was a loop or a bouncing trunk somewhere.Currently Studying: CCIP - 642-611 - MPLS
Occupation: Tier II NOC Tech - Centurylink
CCIP Progress: [x] BSCI
[x] BGP
[ ] MPLS
[ ] QoS -
phantasm
Member Posts: 995
I agree with all that has been said in this thread, wolverine13 has outlined the issue and provided some good information (wouldn't expect less from him). As a side note I will provide this link for some after hours reading:
http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a008015bfd6.shtml"No man ever steps in the same river twice, for it's not the same river and he's not the same man." -Heraclitus -
creamy_stew
Member Posts: 406 ■■■□□□□□□□
wolverene13 wrote: »Internet-facing interface? If that interface is the same one that you are seeing the interface resets on.
No, it isn't. I switched routers somewhere midstream, sorry When I talk about the DR(The one with Internet connectivity) I'm just using it to show a different point of view.
The problem router ("hostname somerouter", not DR/BDR) is the one causing all the OSPF logs I showed on the DESIGNATED-ROUTER01.
edit:somerouter is also the one with the interface resets. It' quarter past midnight here, so I don't have the stamina to jump through the hoops required to get the logs today. -
creamy_stew
Member Posts: 406 ■■■□□□□□□□
I agree with all that has been said in this thread, wolverine13 has outlined the issue and provided some good information (wouldn't expect less from him). As a side note I will provide this link for some after hours reading:
Troubleshooting Switch Port and Interface Problems - Cisco Systems
wolverine13 has certainly provided lots of great feedback.
