Interface resets on an eth interface?

creamy_stewcreamy_stew Member Posts: 406 ■■■□□□□□□□
What, if anything, are Interface Resets indicative of on an ethernet interface?

Usually, I see perhaps a couple on an interface whose counters haven't been reset for a long time. Now, I'm having sporadic but major problems with an c1812. It loses OSPF connections to it's neighbors, sometimes it seems it just loses the network connection altogether.

Initially, I just looked at whether the interface had gone down, CRC errors FCS etc, but no problem there. Later, I noticed that the "interface resets" counter in sh int increases at a rediculous rate. What info I can find online only refers to serial interfaces and I don't really understand what they mean anyway. One comment seemed to indicate that runts are somehow related to them, but I get no runts at at all.

sh int output for your enjoyment:

FastEthernet0 is up, line protocol is up
Hardware is PQ3_TSEC, address is 001f.9e8c.xxxx (bia 001f.9e8c.xxxx)
Internet address is 10.x.y.z/25
MTU 1500 bytes, BW 10000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3000 bits/sec, 3 packets/sec
5 minute output rate 2000 bits/sec, 2 packets/sec
1011966 packets input, 367334058 bytes
Received 105294 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
983881 packets output, 76858816 bytes, 0 underruns
0 output errors, 0 collisions, 9453 interface resets <---- Holy cow!
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Itchy... Tasty!
[X] DCICN
[X] IINS

[ ] CCDA
[ ] DCICT

Comments

  • networker050184networker050184 Mod Posts: 11,962 Mod
    What is it connected to and whats the config look like?
    An expert is a man who has made all the mistakes which can be made.
  • Forsaken_GAForsaken_GA Member Posts: 4,024
    check the duplex on the other end. And try a different cable.
  • chmorinchmorin Member Posts: 1,446 ■■■■■□□□□□
    check the duplex on the other end. And try a different cable.

    Duplex issues would usually be throwing output errors I think... But I would still confirm that both sides are agreeing on the speed and duplex configurations.
    Currently Pursuing
    WGU (BS in IT Network Administration) - 52%| CCIE:Voice Written - 0% (0/200 Hours)
    mikej412 wrote:
    Cisco Networking isn't just a job, it's a Lifestyle.
  • Forsaken_GAForsaken_GA Member Posts: 4,024
    chmorin wrote: »
    Duplex issues would usually be throwing output errors I think... But I would still confirm that both sides are agreeing on the speed and duplex configurations.

    It would, but I've seen noisy servers with duplex mismatches cause interface resets before (not at the rate shown in his output though).

    duplex, along with dns, is one of the things I always check when things start breaking, better to rule it out early than find out it's the cause a couple hours down the road.

    More likely than not, he's got a bad cable or transceiver
  • creamy_stewcreamy_stew Member Posts: 406 ■■■□□□□□□□
    What is it connected to and whats the config look like?

    In a broader sense, it connects to a leased point-to-multipoint VLAN. The handoff switch is usually a cisco or extreme networks, though I haven't seen it with my own eyes. Also, the plot just thickened: Between the provider switch and this router, there's single-mode fiber with a media converter at each end. No-one seems to know who actually owns the converters icon_sad.gif

    Anyway: sh run brie (hope I removed all sensitive info)

    Current configuration : 10293 bytes
    !
    ! No configuration change since last restart
    !
    version 12.4
    no service pad
    service tcp-keepalives-in
    service tcp-keepalives-out
    service timestamps debug datetime localtime
    service timestamps log datetime localtime
    service password-encryption
    service sequence-numbers
    !
    hostname somerouter
    !
    boot-start-marker
    boot-end-marker
    !
    logging buffered 51200 informational
    !
    aaa new-model
    !
    !
    aaa group server radius RadiusServers
    server 10.22.0.10 auth-port 1812 acct-port 1813
    server 10.22.0.11 auth-port 1812 acct-port 1813
    !
    aaa authentication login default group RadiusServers local
    aaa authorization exec default group RadiusServers local
    !
    !
    aaa session-id common
    clock timezone Berlin 1
    clock summer-time Berlin recurring last Sun Mar 2:00 last Sun Oct 3:00
    !
    <snip crypto>
    !
    !
    ip cef
    no ip dhcp use vrf connected
    ip dhcp excluded-address 10.22.7.1 10.22.7.19
    ip dhcp excluded-address 10.22.27.1 10.22.27.19
    !
    ip dhcp pool dhcp-pool
    network 10.22.7.0 255.255.255.0
    domain-name somedomain.se
    dns-server 10.22.0.10 10.22.0.11
    netbios-name-server 10.22.0.10 10.22.0.11
    netbios-node-type p-node
    default-router 10.22.7.1
    lease 0 2
    !
    ip dhcp pool voip-pool
    network 10.22.27.0 255.255.255.0
    domain-name somedomain.se
    dns-server 10.22.0.10 10.22.0.11
    default-router 10.22.27.1
    !
    !
    no ip domain lookup
    ip domain name somedomain.se
    ip auth-proxy max-nodata-conns 3
    ip admission max-nodata-conns 3
    !
    multilink bundle-name authenticated
    !
    !
    !
    spanning-tree uplinkfast
    spanning-tree backbonefast
    vtp domain somedomain
    vtp mode transparent
    <snip users>
    !
    !
    archive
    log config
    hidekeys
    !
    !
    vlan 20
    name VOIP
    !
    !
    <snip policy/classmaps>
    !
    !
    !
    !
    interface Loopback0
    ip address 10.22.16.136 255.255.255.255
    !
    interface FastEthernet0
    ip address 10.22.16.9 255.255.255.128
    speed auto
    full-duplex
    service-policy output SHAPER
    !
    interface FastEthernet1
    no ip address
    shutdown
    duplex auto
    speed auto
    !
    interface BRI0
    no ip address
    encapsulation hdlc
    shutdown
    !
    interface FastEthernet2
    switchport trunk allowed vlan 1,20,1002-1005
    switchport mode trunk
    !
    interface FastEthernet3
    switchport trunk allowed vlan 1,20,1002-1005
    switchport mode trunk
    spanning-tree portfast
    !
    interface FastEthernet4
    switchport trunk allowed vlan 1,20,1002-1005
    switchport mode trunk
    spanning-tree portfast
    !
    interface FastEthernet5
    switchport trunk allowed vlan 1,20,1002-1005
    switchport mode trunk
    spanning-tree portfast
    !
    interface FastEthernet6
    switchport trunk allowed vlan 1,20,1002-1005
    switchport mode trunk
    spanning-tree portfast
    !
    interface FastEthernet7
    switchport trunk allowed vlan 1,20,1002-1005
    switchport mode trunk
    spanning-tree portfast
    !
    interface FastEthernet8
    switchport trunk allowed vlan 1,20,1002-1005
    switchport mode trunk
    spanning-tree portfast
    !
    interface FastEthernet9
    switchport trunk allowed vlan 1,20,1002-1005
    switchport mode trunk
    spanning-tree portfast
    !
    interface Vlan1
    description $ETH-SW-LAUNCH$$INTF-INFO-FE 2$
    ip address 10.22.7.1 255.255.255.0
    ip tcp adjust-mss 1452
    !
    interface Vlan20
    description VOIP
    ip address 10.22.27.1 255.255.255.0
    ip access-group VOIP-traffic in
    no ip redirects
    no ip unreachables
    no ip proxy-arp
    ip route-cache flow
    ip tcp adjust-mss 1452
    no autostate
    !
    router ospf 1
    log-adjacency-changes
    passive-interface FastEthernet1
    passive-interface Vlan1
    passive-interface Vlan20
    network 10.22.7.0 0.0.0.255 area 0
    network 10.22.16.0 0.0.0.127 area 0
    network 10.22.16.136 0.0.0.0 area 0
    network 10.22.27.0 0.0.0.255 area 0
    !
    ip forward-protocol nd
    !
    !
    ip http server
    ip http access-class 23
    ip http authentication local
    ip http secure-server
    ip http timeout-policy idle 60 life 86400 requests 10000
    !
    <snip acls>
    snmp-server community <snip>
    snmp-server ifindex persist
    snmp-server trap-source Loopback0
    snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
    snmp-server enable traps vrrp
    snmp-server enable traps tty
    snmp-server enable traps eigrp
    snmp-server enable traps flash insertion removal
    snmp-server enable traps isdn call-information
    snmp-server enable traps isdn layer2
    snmp-server enable traps isdn chan-not-avail
    snmp-server enable traps isdn ietf
    snmp-server enable traps envmon
    snmp-server enable traps disassociate
    snmp-server enable traps deauthenticate
    snmp-server enable traps authenticate-fail
    snmp-server enable traps dot11-qos
    snmp-server enable traps switch-over
    snmp-server enable traps rogue-ap
    snmp-server enable traps wlan-wep
    snmp-server enable traps aaa_server
    snmp-server enable traps atm subif
    snmp-server enable traps bgp
    snmp-server enable traps bulkstat collection transfer
    snmp-server enable traps cnpd
    snmp-server enable traps config-copy
    snmp-server enable traps config
    snmp-server enable traps entity
    snmp-server enable traps resource-policy
    snmp-server enable traps event-manager
    snmp-server enable traps frame-relay
    snmp-server enable traps frame-relay subif
    snmp-server enable traps hsrp
    snmp-server enable traps ipmulticast
    snmp-server enable traps mpls ldp
    snmp-server enable traps mpls traffic-eng
    snmp-server enable traps mpls vpn
    snmp-server enable traps msdp
    snmp-server enable traps mvpn
    snmp-server enable traps ospf state-change
    snmp-server enable traps ospf errors
    snmp-server enable traps ospf retransmit
    snmp-server enable traps ospf lsa
    snmp-server enable traps ospf cisco-specific state-change nssa-trans-change
    snmp-server enable traps ospf cisco-specific state-change shamlink interface-old
    snmp-server enable traps ospf cisco-specific state-change shamlink neighbor
    snmp-server enable traps ospf cisco-specific errors
    snmp-server enable traps ospf cisco-specific retransmit
    snmp-server enable traps ospf cisco-specific lsa
    snmp-server enable traps pim neighbor-change rp-mapping-change invalid-pim-message
    snmp-server enable traps pppoe
    snmp-server enable traps cpu threshold
    snmp-server enable traps rsvp
    snmp-server enable traps ipsla
    snmp-server enable traps syslog
    snmp-server enable traps l2tun session
    snmp-server enable traps l2tun pseudowire status
    snmp-server enable traps pw vc
    snmp-server enable traps firewall serverstatus
    snmp-server enable traps isakmp policy add
    snmp-server enable traps isakmp policy delete
    snmp-server enable traps isakmp tunnel start
    snmp-server enable traps isakmp tunnel stop
    snmp-server enable traps ipsec cryptomap add
    snmp-server enable traps ipsec cryptomap delete
    snmp-server enable traps ipsec cryptomap attach
    snmp-server enable traps ipsec cryptomap detach
    snmp-server enable traps ipsec tunnel start
    snmp-server enable traps ipsec tunnel stop
    snmp-server enable traps ipsec too-many-sas
    snmp-server host <snip>
    snmp-server host <snip>
    no cdp run
    !
    !
    !
    !
    !
    radius-server host 10.22.0.10 auth-port 1812 acct-port 1813 key 7 <snip>
    radius-server host 10.22.0.11 auth-port 1812 acct-port 1813 key 7 <snip>
    radius-server vsa send authentication
    !
    control-plane
    !
    <snip banner>
    !
    line con 0
    logging synchronous
    line aux 0
    line vty 0 4
    access-class 23 in
    privilege level 15
    logging synchronous
    transport input ssh
    line vty 5 15
    access-class 23 in
    privilege level 15
    logging synchronous
    transport input ssh
    !
    ntp clock-period 17180089
    ntp server 10.22.16.1 prefer
    ntp server 10.22.16.2
    end
    Itchy... Tasty!
    [X] DCICN
    [X] IINS

    [ ] CCDA
    [ ] DCICT
  • Forsaken_GAForsaken_GA Member Posts: 4,024
    In a broader sense, it connects to a leased point-to-multipoint VLAN. The handoff switch is usually a cisco or extreme networks, though I haven't seen it with my own eyes. Also, the plot just thickened: Between the provider switch and this router, there's single-mode fiber with a media converter at each end. No-one seems to know who actually owns the converters icon_sad.gif

    I'd bet dollars to donuts the media converter is the root cause. Hate those damned things.
  • networker050184networker050184 Mod Posts: 11,962 Mod
    First place I'd start is the converters. You are probably losing link on the fiber but not the physical link to the converter.
    An expert is a man who has made all the mistakes which can be made.
  • creamy_stewcreamy_stew Member Posts: 406 ■■■□□□□□□□
    First place I'd start is the converters. You are probably losing link on the fiber but not the physical link to the converter.

    Oh, right! Some converters keep the link up even if the link on the other side dies. That must make for some dynamic routing fun :)

    Could this be the cause of the interface resets? I still don't get what they actually mean :)
    Itchy... Tasty!
    [X] DCICN
    [X] IINS

    [ ] CCDA
    [ ] DCICT
  • HeeroHeero Member Posts: 486
    Oh, right! Some converters keep the link up even if the link on the other side dies. That must make for some dynamic routing fun :)

    Could this be the cause of the interface resets? I still don't get what they actually mean :)

    I've experienced the faulty link loss detection with a CWDM setup before. It is quite annoying, but its a good place to start.
  • Forsaken_GAForsaken_GA Member Posts: 4,024
    Oh, right! Some converters keep the link up even if the link on the other side dies. That must make for some dynamic routing fun :)

    Could this be the cause of the interface resets? I still don't get what they actually mean :)

    it basically means that the interface had packets all queued up but they didn't get transmitted, so it resets the interface
  • creamy_stewcreamy_stew Member Posts: 406 ■■■□□□□□□□
    it basically means that the interface had packets all queued up but they didn't get transmitted, so it resets the interface

    So how does that happen? If the interface is up, why doesn't it just spew out those eth frames? I think I'm about to get one of those elusive "Aha" moments :)
    Itchy... Tasty!
    [X] DCICN
    [X] IINS

    [ ] CCDA
    [ ] DCICT
  • Forsaken_GAForsaken_GA Member Posts: 4,024
    So how does that happen? If the interface is up, why doesn't it just spew out those eth frames? I think I'm about to get one of those elusive "Aha" moments :)

    I'm not entirely sure when it comes to ethernet. I know for serial interfaces, it's because the interface has it's doubts about the line protocol actually being up, so I'd imagine it to be something similar.
  • wolverene13wolverene13 Member Posts: 87 ■■□□□□□□□□
    Oh, right! Some converters keep the link up even if the link on the other side dies. That must make for some dynamic routing fun :)

    Could this be the cause of the interface resets? I still don't get what they actually mean :)

    Interface resets indicate that the port bounced. Basically you have a bouncing circuit. Media converters are circuit equipment and owned by the provider. I work for a large WAN provider and we have problems with those stupid media converters all the time (usually Omnitrons). I would first open a trouble ticket with them and then get real pushy and request to be moved to straight fiber.
    Currently Studying: CCIP - 642-611 - MPLS
    Occupation: Tier II NOC Tech - Centurylink
    CCIP Progress: [x] BSCI
    [x] BGP
    [ ] MPLS
    [ ] QoS
  • creamy_stewcreamy_stew Member Posts: 406 ■■■□□□□□□□
    Interface resets indicate that the port bounced. Basically you have a bouncing circuit. Media converters are circuit equipment and owned by the provider. I work for a large WAN provider and we have problems with those stupid media converters all the time (usually Omnitrons). I would first open a trouble ticket with them and then get real pushy and request to be moved to straight fiber.

    Hm, I thought a bounce on an ethernet interface was were the port loses the link? This isn't happening here. There are around double digits worth of interface resets per second. However, I see no Link up/down/flaps. The provider didn't see any up/downs either.

    Do you mean the fiber interfaces on the converters are flapping, and those flaps, in turn, cause the resets on my router?
    Itchy... Tasty!
    [X] DCICN
    [X] IINS

    [ ] CCDA
    [ ] DCICT
  • Forsaken_GAForsaken_GA Member Posts: 4,024
    Hm, I thought a bounce on an ethernet interface was were the port loses the link? This isn't happening here. There are around double digits worth of interface resets per second. However, I see no Link up/down/flaps. The provider didn't see any up/downs either.

    Do you mean the fiber interfaces on the converters are flapping, and those flaps, in turn, cause the resets on my router?

    If the fiber converter is munged and isn't, you know, converting, both sides of the physical link can still appear to be up, so you wouldn't see any link up/down messages.
  • wolverene13wolverene13 Member Posts: 87 ■■□□□□□□□□
    Hm, I thought a bounce on an ethernet interface was were the port loses the link? This isn't happening here. There are around double digits worth of interface resets per second. However, I see no Link up/down/flaps. The provider didn't see any up/downs either.

    Do you mean the fiber interfaces on the converters are flapping, and those flaps, in turn, cause the resets on my router?

    Unless your provider has link propagation turned on in the media converter, neither of you will see a bounce if it happens on the fiber itself. Both you and the carrier will only be able to see events that occur between the media converter and your respective devices. When your router sees the link as being "up," it is only seeing it's connection as up to the media converter and is not actually seeing the link as being up to the provider's device on the other end of the circuit. Your provider will see the same thing on their end. I would have your carrier run head to head testing on the circuit to either prove or refute this assumption. This is going to require a tech in the CO and a tech at your site, both with test sets. They will then push traffic to each others' test sets and be able to determine if there is an issue. However, normally if they don't find an issue after doing this, you'll get billed for the dispatch. You won't get a bill if they actually find a problem on their end (or at least that's how it works with my company). So, I would thoroughly check your end before requesting that they do this. Another thing; are you actually seeing a problem as a result of this, or are you just noticing the interface resets and want to know why this is showing up? If no problem is arising as a result of the resets, it may just be a cosmetic bug in the version of IOS you are running.
    Currently Studying: CCIP - 642-611 - MPLS
    Occupation: Tier II NOC Tech - Centurylink
    CCIP Progress: [x] BSCI
    [x] BGP
    [ ] MPLS
    [ ] QoS
  • creamy_stewcreamy_stew Member Posts: 406 ■■■□□□□□□□
    Unless your provider has link propagation turned on in the media converter, neither of you will see a bounce if it happens on the fiber itself. Both you and the carrier will only be able to see events that occur between the media converter and your respective devices. When your router sees the link as being "up," it is only seeing it's connection as up to the media converter and is not actually seeing the link as being up to the provider's device on the other end of the circuit. Your provider will see the same thing on their end. I would have your carrier run head to head testing on the circuit to either prove or refute this assumption. This is going to require a tech in the CO and a tech at your site, both with test sets. They will then push traffic to each others' test sets and be able to determine if there is an issue. However, normally if they don't find an issue after doing this, you'll get billed for the dispatch. You won't get a bill if they actually find a problem on their end (or at least that's how it works with my company). So, I would thoroughly check your end before requesting that they do this. Another thing; are you actually seeing a problem as a result of this, or are you just noticing the interface resets and want to know why this is showing up? If no problem is arising as a result of the resets, it may just be a cosmetic bug in the version of IOS you are running.

    About seeing problems, here's an OSPF log from the designated router from when the kaka hit the fan (10.22.16.136 is the router I was talking about):

    edit: Curiously, I didn't get any alarms from 10.22.16.136 until approx. 12icon_mad.gifx. This seems to suggest, to me, that they had a very low bandwidth loop that was picking up speed. and OSPF was able to converge fast enough for the polling not to notice before that time?

    edit2: I really should learn how to make those neat in-post scrolling frames :)
    I just never posted logs/configs until now.



    DESIGNATED-ROUTER01#sh logg | i OSPF
    097566: Sep 29 11:08:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.132 on FastEthernet0 from FULL to DOWN, Neighbor Down: Dead timer expired
    097567: Sep 29 11:09:01: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.132 on FastEthernet0 from LOADING to FULL, Loading Done
    097573: Sep 29 11:15:11: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.142 on FastEthernet0 from FULL to DOWN, Neighbor Down: Dead timer expired
    097574: Sep 29 11:16:11: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.142 on FastEthernet0 from LOADING to FULL, Loading Done
    097575: Sep 29 11:19:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097576: Sep 29 11:20:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097577: Sep 29 11:20:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097578: Sep 29 11:21:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097579: Sep 29 11:22:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097597: Sep 29 11:57:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097599: Sep 29 11:58:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097609: Sep 29 11:59:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097610: Sep 29 12:00:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097615: Sep 29 12:00:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097645: Sep 29 12:01:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097646: Sep 29 12:02:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097647: Sep 29 12:03:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097648: Sep 29 12:04:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097653: Sep 29 12:05:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097654: Sep 29 12:06:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097655: Sep 29 12:07:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097656: Sep 29 12:08:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097657: Sep 29 12:08:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097658: Sep 29 12:09:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097659: Sep 29 12:10:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097660: Sep 29 12:11:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097661: Sep 29 12:12:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097662: Sep 29 12:13:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097663: Sep 29 12:14:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097664: Sep 29 12:14:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097665: Sep 29 12:15:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097666: Sep 29 12:16:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097667: Sep 29 12:18:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097668: Sep 29 12:19:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097669: Sep 29 12:20:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097671: Sep 29 12:21:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097672: Sep 29 12:21:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097673: Sep 29 12:22:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097674: Sep 29 12:24:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097675: Sep 29 12:25:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097676: Sep 29 12:26:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097677: Sep 29 12:26:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097678: Sep 29 12:27:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097679: Sep 29 12:28:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097680: Sep 29 12:29:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097681: Sep 29 12:30:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097682: Sep 29 12:31:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097683: Sep 29 12:32:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097684: Sep 29 12:32:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097685: Sep 29 12:33:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097686: Sep 29 12:34:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097687: Sep 29 12:35:37: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097688: Sep 29 12:36:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097689: Sep 29 12:37:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097690: Sep 29 12:38:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097691: Sep 29 12:39:27: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097692: Sep 29 12:40:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097693: Sep 29 12:41:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097694: Sep 29 12:41:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097695: Sep 29 12:42:47: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097696: Sep 29 12:43:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097697: Sep 29 12:45:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097698: Sep 29 12:46:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097699: Sep 29 12:46:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097701: Sep 29 12:48:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097702: Sep 29 12:49:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097703: Sep 29 12:50:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097704: Sep 29 12:51:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097736: Sep 29 12:52:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097737: Sep 29 12:53:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097738: Sep 29 12:54:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097739: Sep 29 12:54:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097740: Sep 29 13:05:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097741: Sep 29 13:06:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097743: Sep 29 13:07:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097744: Sep 29 13:08:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097746: Sep 29 13:08:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097747: Sep 29 13:09:48: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097748: Sep 29 13:10:38: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097749: Sep 29 13:11:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097750: Sep 29 13:12:48: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097751: Sep 29 13:13:38: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097752: Sep 29 13:14:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097753: Sep 29 13:15:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097754: Sep 29 13:16:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097755: Sep 29 13:16:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097756: Sep 29 13:17:48: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097757: Sep 29 13:18:38: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097758: Sep 29 13:19:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097763: Sep 29 13:20:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097764: Sep 29 13:21:38: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097765: Sep 29 13:22:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097766: Sep 29 13:23:48: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097767: Sep 29 13:24:38: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097768: Sep 29 13:25:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097770: Sep 29 13:26:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097771: Sep 29 13:27:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097772: Sep 29 13:28:08: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097774: Sep 29 13:29:28: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097776: Sep 29 13:30:18: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097790: Sep 29 13:54:39: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from FULL to DOWN, Neighbor Down: Dead timer expired
    097791: Sep 29 13:55:10: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    Itchy... Tasty!
    [X] DCICN
    [X] IINS

    [ ] CCDA
    [ ] DCICT
  • wolverene13wolverene13 Member Posts: 87 ■■□□□□□□□□
    About seeing problems, here's an OSPF log from the designated router from when the kaka hit the fan (10.22.16.136 is the router I was talking about):

    edit: Curiously, I didn't get any alarms from 10.22.16.136 until approx. 12icon_mad.gifx. This seems to suggest, to me, that they had a very low bandwidth loop that was picking up speed. and OSPF was able to converge fast enough for the polling not to notice before that time?

    edit2: I really should learn how to make those neat in-post scrolling frames :)
    I just never posted logs/configs until now.



    DESIGNATED-ROUTER01#sh logg | i OSPF
    097566: Sep 29 11:08:58: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.132 on FastEthernet0 from FULL to DOWN, Neighbor Down: Dead timer expired
    097567: Sep 29 11:09:01: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.132 on FastEthernet0 from LOADING to FULL, Loading Done
    097573: Sep 29 11:15:11: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.142 on FastEthernet0 from FULL to DOWN, Neighbor Down: Dead timer expired
    097574: Sep 29 11:16:11: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.142 on FastEthernet0 from LOADING to FULL, Loading Done
    097575: Sep 29 11:19:17: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097576: Sep 29 11:20:07: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done
    097577: Sep 29 11:20:57: %OSPF-5-ADJCHG: Process 1, Nbr 10.22.16.136 on FastEthernet0 from LOADING to FULL, Loading Done

    I noticed this is the output from the log with everything that has the word "OSPF" in it. Are there any other pertinent events occurring too (like other logs without "OSPF" in them)? Another thing that I noticed is that OSPF is only going from LOADING to FULL most of the time and not really going down. This indicates a logical issue if that's the case (it still may be resulting from a physical problem, but it's logical in nature because all that is happening is that the two routers are updating their routing tables based on the information they have provided one another). Another thing is that the log entries are exactly 50 seconds apart (with the exception of the ones in the beginning). STP takes 50 seconds to converge, so a Layer 2 loop is entirely possible. Do you happen to know what type of network your provider has you on? Is it Metro Ethernet, ATM, MPLS, or Frame?
    Currently Studying: CCIP - 642-611 - MPLS
    Occupation: Tier II NOC Tech - Centurylink
    CCIP Progress: [x] BSCI
    [x] BGP
    [ ] MPLS
    [ ] QoS
  • phoeneousphoeneous Member Posts: 2,333 ■■■■■■■□□□
    Basically you have a bouncing circuit.

    That's what it sounds like to me. Anything good in sh log?
  • creamy_stewcreamy_stew Member Posts: 406 ■■■□□□□□□□
    Another thing is that the log entries are exactly 50 seconds apart (with the exception of the ones in the beginning). STP takes 50 seconds to converge, so a Layer 2 loop is entirely possible. Do you happen to know what type of network your provider has you on? Is it Metro Ethernet, ATM, MPLS, or Frame?

    Well, this (the DR) also NATs some traffic (it announces a default route) (so you get all sorts of interesting logs on the Internet-facing interface, and it also logs some acls, so the "i OSPF" was strictly a courtesy :))

    I also did an " i include" for both the actual interface and the loopback(neighbor) interface. That didn't provide any more info. If there's a particular time period you're intereted in, I can dig into the syslogs and give give you the whole shebang (It's just a pain to look through them to verify that there's no senistive info leaking out:).

    As I said I think they had a loop, I also believe they are running eth somewhere between "us" and "them".
    Itchy... Tasty!
    [X] DCICN
    [X] IINS

    [ ] CCDA
    [ ] DCICT
  • creamy_stewcreamy_stew Member Posts: 406 ■■■□□□□□□□
    phoeneous wrote: »
    That's what it sounds like to me. Anything good in sh log?

    Nothing really good, AFAICT.

    The outward facing interface never loses link

    The OSPF process times out all or most of the other routers

    On the DR, I can see it going from LOADINg to FULL a gazillion times, occasioanlly going from FULL to DOWN (from memory).

    Apart from that, everything semms hunky-dory(?)

    This is obviously an STP problem somwhere along the path. What I don't get is those interface resets. Do they mean I should fix something?
    Itchy... Tasty!
    [X] DCICN
    [X] IINS

    [ ] CCDA
    [ ] DCICT
  • wolverene13wolverene13 Member Posts: 87 ■■□□□□□□□□
    Well, this (the DR) also NATs some traffic (it announces a default route) (so you get all sorts of interesting logs on the Internet-facing interface, and it also logs some acls, so the "i OSPF" was strictly a courtesy :))

    Internet-facing interface? If that interface is the same one that you are seeing the interface resets on, is this OSPF relationship over a GRE tunnel or something? If that's the case, there are about 1,000 things that could be happening. Did you do any traceroutes when this was going on (or is it still happening)?
    I also did an " i include" for both the actual interface and the loopback(neighbor) interface. That didn't provide any more info. If there's a particular time period you're intereted in, I can dig into the syslogs and give give you the whole shebang (It's just a pain to look through them to verify that there's no senistive info leaking out:).

    I think a snapshot of the logs starting with the entry that is right after one that references OSPF going from LOADING to FULL all the way until just before the next OSPF LOADING/FULL entry would be helpful. I think seeing the logs from right after the LOADING/FULL entry to the FULL to DOWN would also show something useful. Obviously you should omit anything you don't want others to see.
    As I said I think they had a loop, I also believe they are running eth somewhere between "us" and "them".

    If they are doing QinQ tunneling at all, I would almost bet my next paycheck this was a loop or a bouncing trunk somewhere.
    Currently Studying: CCIP - 642-611 - MPLS
    Occupation: Tier II NOC Tech - Centurylink
    CCIP Progress: [x] BSCI
    [x] BGP
    [ ] MPLS
    [ ] QoS
  • phantasmphantasm Member Posts: 995
    I agree with all that has been said in this thread, wolverine13 has outlined the issue and provided some good information (wouldn't expect less from him). As a side note I will provide this link for some after hours reading:

    http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a008015bfd6.shtml
    "No man ever steps in the same river twice, for it's not the same river and he's not the same man." -Heraclitus
  • creamy_stewcreamy_stew Member Posts: 406 ■■■□□□□□□□
    Internet-facing interface? If that interface is the same one that you are seeing the interface resets on.

    No, it isn't. I switched routers somewhere midstream, sorry :) When I talk about the DR(The one with Internet connectivity) I'm just using it to show a different point of view.

    The problem router ("hostname somerouter", not DR/BDR) is the one causing all the OSPF logs I showed on the DESIGNATED-ROUTER01.

    edit:somerouter is also the one with the interface resets. It' quarter past midnight here, so I don't have the stamina to jump through the hoops required to get the logs today.
    Itchy... Tasty!
    [X] DCICN
    [X] IINS

    [ ] CCDA
    [ ] DCICT
  • creamy_stewcreamy_stew Member Posts: 406 ■■■□□□□□□□
    phantasm wrote: »
    I agree with all that has been said in this thread, wolverine13 has outlined the issue and provided some good information (wouldn't expect less from him). As a side note I will provide this link for some after hours reading:

    Troubleshooting Switch Port and Interface Problems - Cisco Systems

    wolverine13 has certainly provided lots of great feedback.
    Itchy... Tasty!
    [X] DCICN
    [X] IINS

    [ ] CCDA
    [ ] DCICT
Sign In or Register to comment.