High CPU Utilization on Cisco 4506-E

razamrazam Member Posts: 39 ■■□□□□□□□□
Hello all,

i have a cisco core switch 4506-E, its processor utilization is very high, 50% on average.

after checking the output of "show process cpu detail" got to know that it is because of "ARP Input" process.

anyone can suggest on how to solve this issue ? please see below the image



  • FloOzFloOz Member Posts: 1,614 ■■■■□□□□□□
    do you have a static default route pointing towards and exit interface?
    Example- ip route ser0/0
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    static route is
    ip route 172.x.x.x
  • FloOzFloOz Member Posts: 1,614 ■■■■□□□□□□
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    thanks for the share but this document only explains the arp procedure and it will cause high traffic, doesnt give any solution that how can we overcome this problem.
  • networker050184networker050184 Mod Posts: 11,962 Mod
    You need to find and fix the arp issue. Do you know how arp works and how it could become a problem?
    An expert is a man who has made all the mistakes which can be made.
  • MonkerzMonkerz Member Posts: 842
    Can you include the following two outputs from your 4506?

    show run | inc ip route
    show ip arp
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    core-switch#show running-config | include ip route
    ip route
    ip route GigabitEthernet2/34
    ip route
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    show ip arp

    #show ip arp
    Protocol Address Age (min) Hardware Addr Type Interface
    Internet - 6c20.56cd.7eff ARPA Vlan51
    Internet 2 7c61.9398.e795 ARPA Vlan51
    Internet 1 5cf8.a1fa.fdda ARPA Vlan51
    Internet 0 e0ca.940b.266d ARPA Vlan51
    Internet 101 0023.b1aa.435d ARPA Vlan51
    Internet 3 f81a.67ce.c861 ARPA Vlan51
    Internet 2 a0f3.c19a.da36 ARPA Vlan51
    Internet 2 6470.026e.5a9b ARPA Vlan51
    Internet 0 e8e0.b741.066a ARPA Vlan51
    Internet 78 0015.c5b8.c007 ARPA Vlan51
    Internet 0 1803.739f.43bd ARPA Vlan51
    Internet 0 b8ff.6169.7c9f ARPA Vlan51
    Internet 0 7845.c4a2.1c9d ARPA Vlan51
    Internet 7 b005.9473.1a9d ARPA Vlan51
    Internet 126 0026.756f.899f ARPA Vlan51
    Internet 202 0026.756f.8417 ARPA Vlan51
    Internet 30 0026.753b.788e ARPA Vlan51
    Internet 0 7845.c4a5.d061 ARPA Vlan51
    Internet 0 e0db.55d5.d22f ARPA Vlan51
    Internet 224 7845.c4ae.f796 ARPA Vlan51
    Internet 0 e803.9aed.aa5e ARPA Vlan51
    Internet 155 a0b3.cc7f.3bb3 ARPA Vlan51
    Internet 0 0026.756f.8b58 ARPA Vlan51
    Internet - 6c20.56cd.7eff ARPA Vlan52
    Internet 1 5c95.ae29.b081 ARPA Vlan52
    Internet 58 5cf8.a14a.6443 ARPA Vlan52
    Internet 5 54e6.fccb.d859 ARPA Vlan52
    Internet 82 0037.6def.d920 ARPA Vlan52
    Internet 236 3c07.543e.1640 ARPA Vlan52
    Internet 73 dc0e.a167.58f2 ARPA Vlan52
    Internet 2 10dd.b1a4.7b80 ARPA Vlan52
    Internet 4 a0f3.c19f.e8eb ARPA Vlan52
    Internet 1 a0f3.c1db.0d85 ARPA Vlan52
    Internet 3 a0f3.c166.1571 ARPA Vlan52
    Internet 0 e8e0.b7e1.8864 ARPA Vlan52
    Internet 0 68a3.c47c.fc7d ARPA Vlan52
    Internet 179 0026.756f.8d20 ARPA Vlan52
    Internet 0 6036.dd3a.b295 ARPA Vlan52
    Internet 184 e89d.87f0.5830 ARPA Vlan52
    Internet 186 d4c9.ef67.0b8b ARPA Vlan52
    Internet 88 e8e0.b768.8764 ARPA Vlan52
    Internet 1 5046.5d49.4e3f ARPA Vlan52
    Protocol Address Age (min) Hardware Addr Type Interface
    Internet 201 68a8.6d29.2ef4 ARPA Vlan52
    Internet 81 b803.05c8.717f ARPA Vlan52
    Internet 169 8832.9b03.d2aa ARPA Vlan52
    Internet 85 a4eb.d384.57e5 ARPA Vlan52
    Internet 47 b878.2e56.b1a6 ARPA Vlan52
    Internet 0 bc3b.af7c.2166 ARPA Vlan52
    Internet 0 5c96.9d89.c901 ARPA Vlan52
    Internet 22 3c97.0e86.1c31 ARPA Vlan52
    Internet 0 0026.753b.8dfa ARPA Vlan52
    Internet 176 20aa.4ba9.adc9 ARPA Vlan52
    Internet 0 c80a.a9d4.1b74 ARPA Vlan52
    Internet 218 0025.6472.aff0 ARPA Vlan52
    Internet 1 2089.84eb.b786 ARPA Vlan52
    Internet 191 9094.e433.f2c3 ARPA Vlan52
    Internet 40 e0db.55d9.740d ARPA Vlan52
    Internet 175 0026.756f.9317 ARPA Vlan52
    Internet 0 98fc.11e8.3c9e ARPA Vlan52
    Internet 0 f04d.a266.f0a6 ARPA Vlan52
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    Internet - 6c20.56cd.7eff ARPA Vlan53
    Internet 22 0026.756b.25da ARPA Vlan53
    Internet 47 0016.d48a.9201 ARPA Vlan53
    Internet 0 bc85.5633.e2bd ARPA Vlan53
    Internet 180 bcf6.85bf.70d3 ARPA Vlan53
    Internet 195 00ff.7037.9d0e ARPA Vlan53
    Internet 0 e89d.872c.901d ARPA Vlan53
    Internet 21 e8e0.b761.8664 ARPA Vlan53
    Internet 0 b876.3f27.fc4f ARPA Vlan53
    Internet 25 e063.e584.0ea6 ARPA Vlan53
    Internet 133 d422.3f2c.97b9 ARPA Vlan53
    Internet 94 9018.7cec.3a03 ARPA Vlan53
    Internet 16 0c14.20d7.fedb ARPA Vlan53
    Internet 32 0090.a9ce.15e5 ARPA Vlan53
    Internet 23 c83a.3510.9428 ARPA Vlan53
    Internet 251 0415.52e9.53e8 ARPA Vlan53
    Internet 5 a0f3.c189.81b7 ARPA Vlan53
    Internet 168 7845.c4ae.fc37 ARPA Vlan53
    Internet 250 8853.2e56.617f ARPA Vlan53
    Internet 34 e8e0.b718.675b ARPA Vlan53
    Internet 23 c83a.352c.99e0 ARPA Vlan53
    Internet 66 206a.8aee.602f ARPA Vlan53
    Internet 66 94d7.7106.d46f ARPA Vlan53
    Protocol Address Age (min) Hardware Addr Type Interface
    Internet 5 a0f3.c178.df5d ARPA Vlan53
    Internet 0 e89d.87a4.5930 ARPA Vlan53
    Internet 13 a0f3.c1eb.9d82 ARPA Vlan53
    Internet 239 782b.cbd5.9090 ARPA Vlan53
    Internet 3 b4b5.2f7b.5fe0 ARPA Vlan53
    Internet 1 90f6.5224.9971 ARPA Vlan53
    Internet 1 687f.7477.6f48 ARPA Vlan53
    Internet 0 40cb.a82e.0c0c ARPA Vlan53
    Internet 1 d49a.20ea.4438 ARPA Vlan53
    Internet 7 a0f3.c1db.1143 ARPA Vlan53
    Internet 72 98fc.11c2.81b1 ARPA Vlan53
    Internet 3 f8d1.1196.af07 ARPA Vlan53
    Internet 239 905f.2e4e.859a ARPA Vlan53
    Internet 166 a417.313d.5463 ARPA Vlan53
    Internet 53 6431.508c.43ec ARPA Vlan53
    Internet 136 e89d.87da.8d2f ARPA Vlan53
    Internet 144 e063.e565.cc0f ARPA Vlan53
    Internet 151 f8d1.1188.ba2d ARPA Vlan53
    Internet 3 f8d1.1188.ba2d ARPA Vlan53
    Internet 1 88cb.87da.a51f ARPA Vlan53
    Internet 0 b8e8.5672.8a44 ARPA Vlan53
    Internet 43 6021.c00c.f82d ARPA Vlan53
    Internet 0 50cc.f893.0123 ARPA Vlan53
    Internet 1 0c74.c203.526a ARPA Vlan53
    Internet - 6c20.56cd.7eff ARPA Vlan54
    Internet 153 0025.6447.297e ARPA Vlan54
    Internet 76 30f9.edb7.3b5d ARPA Vlan54
    Internet 15 0024.b21a.b7b5 ARPA Vlan54
    Internet 68 848f.69af.4f30 ARPA Vlan54
    Internet 259 4c72.b96b.67a4 ARPA Vlan54
    Internet 31 f4f1.5a98.e0ec ARPA Vlan54
    Internet 143 5cf8.a100.6ab5 ARPA Vlan54
    Internet 1 90f6.526d.5f8d ARPA Vlan54
    Internet 1 a0f3.c1ec.fdc7 ARPA Vlan54
    Internet 2 f4ec.38d4.7e27 ARPA Vlan54
    Internet 4 6470.02fc.5b87 ARPA Vlan54
    Internet 0 90f6.525f.fb95 ARPA Vlan54
    Internet 4 6470.024c.bf35 ARPA Vlan54
    Internet 0 6470.02f0.07cb ARPA Vlan54
    Internet 0 7845.c4a0.d409 ARPA Vlan54
    Internet 186 001e.33a2.54da ARPA Vlan54
    Internet 129 8853.2e8d.d781 ARPA Vlan54
    Internet 0 a0f3.c1db.0961 ARPA Vlan54
    Protocol Address Age (min) Hardware Addr Type Interface
    Internet 0 a45d.3668.c6c0 ARPA Vlan54
    Internet 52 78e7.d1dc.7438 ARPA Vlan54
    Internet 128 7486.7a09.bbd6 ARPA Vlan54
    Internet 4 f8d1.1172.3333 ARPA Vlan54
    Internet 147 b8ac.6f51.0042 ARPA Vlan54
    Internet 3 f8d1.11a9.76cd ARPA Vlan54
    Internet 4 0025.d370.19f5 ARPA Vlan54
    Internet 0 90e6.ba19.b28d ARPA Vlan54
    Internet 0 0024.2c27.5ee9 ARPA Vlan54
    Internet 5 e8e0.b7d5.665b ARPA Vlan54
    Internet 1 f8d1.117a.6d57 ARPA Vlan54
    Internet 116 e8e0.b759.675b ARPA Vlan54
    Internet 223 8853.2e25.61fb ARPA Vlan54
    Internet 185 3cd0.f86f.43b5 ARPA Vlan54
    Internet 140 a0f4.19dc.d176 ARPA Vlan54
    Internet 0 4ceb.4215.22be ARPA Vlan54
    Internet 88 dc0e.a1ef.49b8 ARPA Vlan54
    Internet 0 001e.336f.3306 ARPA Vlan54
    Internet 108 1803.7390.b135 ARPA Vlan54
    Internet 151 e803.9a0b.f115 ARPA Vlan54
    Internet 0 c83a.3512.5245 ARPA Vlan54
    Internet 0 0026.b91b.6c33 ARPA Vlan54
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    it is too long to paste the full output, have shared partial output of show ip arp
  • MonkerzMonkerz Member Posts: 842
    What is within your network? And is a lot of traffic directed to it?
  • Dieg0MDieg0M Member Posts: 861
    Change ip route GigabitEthernet2/34 => to the IP address of next hop instead and check your CPU utilization.
    Follow my CCDE journey at www.routingnull0.com
  • MonkerzMonkerz Member Posts: 842
    Dieg0M wrote: »
    Change ip route GigabitEthernet2/34 => to the IP address of next hop instead and check your CPU utilization.

    I was trying to help to OP understand why this is happening, not just tell him how to correct it.
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    @Monkerz is the WAN ip address. All the traffic is directed to it. assigned on Gig2/34 of my core switch (one with high utilization) assigned on the neighbor device Gig Interface

    will remove ip route GigabitEthernet2/34 ,
    monitor the utilization and share the results.

  • EdTheLadEdTheLad Member Posts: 2,111 ■■■■□□□□□□
    In this case that route pointing to Gi2/34 should have no affect as the is a directly connected subnet.
    It's possible it might be causing an issue for some unknown reason as it's a strange config, it's not required as any traffic destined for .1 will inherently know to go out gi2/34. Remove it see what happens i guess, in your arp outputs alot of addresses have 0 minutes against them, maybe looking at those ip addresses might point towards a common point of failure?
    Networking, sometimes i love it, mostly i hate it.Its all about the $$$$
  • Danielh22185Danielh22185 Member Posts: 1,195 ■■■■□□□□□□
    razam wrote: »
    @Monkerz is the WAN ip address. All the traffic is directed to it. assigned on Gig2/34 of my core switch (one with high utilization) assigned on the neighbor device Gig Interface

    will remove ip route GigabitEthernet2/34 ,
    monitor the utilization and share the results.

    I am just curious why this would cause / or could cause high CPU utilization and how the difference in changing the IP route to the IP of the neighboring device as apposed to the actual physical IP of the connecting interface.
    Currently Studying: IE Stuff...kinda...for now...
    My ultimate career goal: To climb to the top of the computer network industry food chain.
    "Winning means you're willing to go longer, work harder, and give more than anyone else." - Vince Lombardi
  • Dieg0MDieg0M Member Posts: 861
    Ok, so when you put a static route to a neighboring interface, the router sends an ARP request to find the MAC address of the destination network to forward packets, whether the destination is valid or not. Also, the router will receive an ARP response if another router on the broadcast network is responding on behalf of that network (Proxy ARP). This will cause excessive broadcast traffic on the segment. How I can interpret this configuration is that he is sending default traffic to his next hop but in his local routing table he also has a specific route to this address. Because of this, the router will do a recursive lookup of and do ARP requests for all unknown traffic. The best way to configure this would be:
    ip route GigabitEthernet2/34

    This way he will avoid recursive lookups and unnecessary ARP requests. Now in my suggestion I overlooked that his next hop of the static route was infact the same as his default route.
    Follow my CCDE journey at www.routingnull0.com
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    core-switch#show ip arp inspection statistics

    Source Mac Validation : Disabled
    Destination Mac Validation : Disabled
    IP Address Validation : Disabled
    No active or enabled vlans on switch.

    should these be enabled ?

    what is the recommended arp timeout value ?
  • Dieg0MDieg0M Member Posts: 861
    Not unless you have Dynamic Arp Inspection enabled to prevent MAC spoofing. Default is 300 sec and I wouldn't change it unless required to.
    Follow my CCDE journey at www.routingnull0.com
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    please see below the pic, high utilization showing because of another process Cat4k Mgmt LoPri

  • Dieg0MDieg0M Member Posts: 861
    Give us the output of : sh plat heal
    Follow my CCDE journey at www.routingnull0.com
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    "show platform health" output modified

  • Dieg0MDieg0M Member Posts: 861
    Ok check what packets are causing that high cpu: show platform cpu packet statistics
    Follow my CCDE journey at www.routingnull0.com
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    please see the output below for show platform cpu packet statistics

  • Dieg0MDieg0M Member Posts: 861
    Did you change the default route to ip route GigabitEthernet2/34 ? This all seems to indicate excessive ARP lookups.
    Follow my CCDE journey at www.routingnull0.com
  • razamrazam Member Posts: 39 ■■□□□□□□□□

    yesterday monitored the packets received by the Core Switch,

    debug platform packet all receive buffer

    show platform cpu packet buffered

    saw many broadcast packets by two users, traced the ports of those two users and shut down those ports, since then the CPU Utilization is 10%.

    have one question here, i have applied storm control configuration on the access switches, access switch should have put those two ports in an err-disabled state itself.

    please see below my interface configuration on access switches.

    int range fastEthernet 0/1 - 24/48
    description ##TO-END-USERS##
    switchport mode access
    switchport access vlan 54
    speed auto
    duplex auto
    spanning-tree portfast
    spanning-tree bpdufilter enable
    no shut
    no ip dhcp snooping trust
    ip dhcp snooping limit rate 70
    storm-control broadcast level 30.00 10.00
    storm-control action shutdown

    errdisable recovery cause link-flap
    errdisable recovery interval 30

    it should have put the ports with broadcast storm in an err-disabled state.
  • EdTheLadEdTheLad Member Posts: 2,111 ■■■■□□□□□□
    You have the storm control configured for a percentage of bandwidth i.e. 30% for broadcast traffic. So that's allowing 30Mbps of broadcast traffic before the port is shutdown. Lets say the broadcasts are arp requests 46 byte packets.
    30,000,000/(46 * icon_cool.gif = 81521 packets per second.

    That would melt your switch, i'm not sure if your switch supports per packet configuration, but if it does its best practices to use pps values. Bandwidth is never the issue when it comes to the control plane, it's always the number of packets to process that's the problem.
    Networking, sometimes i love it, mostly i hate it.Its all about the $$$$
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    my access switch supports pps, please suggest a value for this.
  • EdTheLadEdTheLad Member Posts: 2,111 ■■■■□□□□□□
    I cant give you a value off the top of my head, i'd have to look at our network and see what traffic is going where, look at stats etc.
    Have a look at the below link for information. Lets say your traffic is 98% unicast and the only broadcast traffic is for arp, then you will have to workout how many end devices are sending arp's to your gateway switch. Looking the your arp cache will give you an idea, remember not all devices send arp's together and this is a packet per second value.
    Internetwork Design Guide -- Broadcasts in Switched LAN Internetworks - DocWiki
    Networking, sometimes i love it, mostly i hate it.Its all about the $$$$
  • razamrazam Member Posts: 39 ■■□□□□□□□□
    solved this issue few days back, now its at normal utilization, controlled the broadcast traffic...

    the time the utilization went high, monitored those pkts, and chked the mac addresses, then after getting to know about the interfaces of access switches where it was coming, saw the input rate n output rate of those interfaces "show interface fa 0/x",

    from the input rate got to know that what value to set for storm control broadcast level.. it has been set at 2.00 rising threshold

    thanks everyone for your help
Sign In or Register to comment.