High CPU Utilization on Cisco 4506-E

razam

Hello all,

i have a cisco core switch 4506-E, its processor utilization is very high, 50% on average.

after checking the output of "show process cpu detail" got to know that it is because of "ARP Input" process.

anyone can suggest on how to solve this issue ? please see below the image

FloOz

do you have a static default route pointing towards and exit interface?
Example- ip route 0.0.0.0 0.0.0.0 ser0/0

razam

yess,
static route is
ip route 0.0.0.0 0.0.0.0 172.x.x.x

FloOz

Check out this page. Troubleshooting High CPU Utilization due to Processes - Cisco Systems

hth

razam

thanks for the share but this document only explains the arp procedure and it will cause high traffic, doesnt give any solution that how can we overcome this problem.

networker050184

You need to find and fix the arp issue. Do you know how arp works and how it could become a problem?

Monkerz

Can you include the following two outputs from your 4506?

show run | inc ip route
show ip arp

razam

core-switch#show running-config | include ip route
ip route 0.0.0.0 0.0.0.0 172.29.201.1
ip route 172.29.201.1 255.255.255.255 GigabitEthernet2/34
ip route 172.80.0.0 255.255.0.0 172.29.200.254

razam

show ip arp

#show ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.29.51.1 - 6c20.56cd.7eff ARPA Vlan51
Internet 172.29.51.7 2 7c61.9398.e795 ARPA Vlan51
Internet 172.29.51.8 1 5cf8.a1fa.fdda ARPA Vlan51
Internet 172.29.51.9 0 e0ca.940b.266d ARPA Vlan51
Internet 172.29.51.15 101 0023.b1aa.435d ARPA Vlan51
Internet 172.29.51.18 3 f81a.67ce.c861 ARPA Vlan51
Internet 172.29.51.19 2 a0f3.c19a.da36 ARPA Vlan51
Internet 172.29.51.22 2 6470.026e.5a9b ARPA Vlan51
Internet 172.29.51.23 0 e8e0.b741.066a ARPA Vlan51
Internet 172.29.51.27 78 0015.c5b8.c007 ARPA Vlan51
Internet 172.29.51.30 0 1803.739f.43bd ARPA Vlan51
Internet 172.29.51.38 0 b8ff.6169.7c9f ARPA Vlan51
Internet 172.29.51.39 0 7845.c4a2.1c9d ARPA Vlan51
Internet 172.29.51.40 7 b005.9473.1a9d ARPA Vlan51
Internet 172.29.51.46 126 0026.756f.899f ARPA Vlan51
Internet 172.29.51.70 202 0026.756f.8417 ARPA Vlan51
Internet 172.29.51.85 30 0026.753b.788e ARPA Vlan51
Internet 172.29.51.140 0 7845.c4a5.d061 ARPA Vlan51
Internet 172.29.51.187 0 e0db.55d5.d22f ARPA Vlan51
Internet 172.29.51.188 224 7845.c4ae.f796 ARPA Vlan51
Internet 172.29.51.210 0 e803.9aed.aa5e ARPA Vlan51
Internet 172.29.51.246 155 a0b3.cc7f.3bb3 ARPA Vlan51
Internet 172.29.51.247 0 0026.756f.8b58 ARPA Vlan51
Internet 172.29.52.1 - 6c20.56cd.7eff ARPA Vlan52
Internet 172.29.52.6 1 5c95.ae29.b081 ARPA Vlan52
Internet 172.29.52.7 58 5cf8.a14a.6443 ARPA Vlan52
Internet 172.29.52.8 5 54e6.fccb.d859 ARPA Vlan52
Internet 172.29.52.10 82 0037.6def.d920 ARPA Vlan52
Internet 172.29.52.12 236 3c07.543e.1640 ARPA Vlan52
Internet 172.29.52.15 73 dc0e.a167.58f2 ARPA Vlan52
Internet 172.29.52.16 2 10dd.b1a4.7b80 ARPA Vlan52
Internet 172.29.52.18 4 a0f3.c19f.e8eb ARPA Vlan52
Internet 172.29.52.19 1 a0f3.c1db.0d85 ARPA Vlan52
Internet 172.29.52.20 3 a0f3.c166.1571 ARPA Vlan52
Internet 172.29.52.23 0 e8e0.b7e1.8864 ARPA Vlan52
Internet 172.29.52.24 0 68a3.c47c.fc7d ARPA Vlan52
Internet 172.29.52.25 179 0026.756f.8d20 ARPA Vlan52
Internet 172.29.52.26 0 6036.dd3a.b295 ARPA Vlan52
Internet 172.29.52.27 184 e89d.87f0.5830 ARPA Vlan52
Internet 172.29.52.28 186 d4c9.ef67.0b8b ARPA Vlan52
Internet 172.29.52.29 88 e8e0.b768.8764 ARPA Vlan52
Internet 172.29.52.30 1 5046.5d49.4e3f ARPA Vlan52
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.29.52.31 201 68a8.6d29.2ef4 ARPA Vlan52
Internet 172.29.52.32 81 b803.05c8.717f ARPA Vlan52
Internet 172.29.52.33 169 8832.9b03.d2aa ARPA Vlan52
Internet 172.29.52.34 85 a4eb.d384.57e5 ARPA Vlan52
Internet 172.29.52.35 47 b878.2e56.b1a6 ARPA Vlan52
Internet 172.29.52.36 0 bc3b.af7c.2166 ARPA Vlan52
Internet 172.29.52.37 0 5c96.9d89.c901 ARPA Vlan52
Internet 172.29.52.38 22 3c97.0e86.1c31 ARPA Vlan52
Internet 172.29.52.39 0 0026.753b.8dfa ARPA Vlan52
Internet 172.29.52.50 176 20aa.4ba9.adc9 ARPA Vlan52
Internet 172.29.52.54 0 c80a.a9d4.1b74 ARPA Vlan52
Internet 172.29.52.58 218 0025.6472.aff0 ARPA Vlan52
Internet 172.29.52.79 1 2089.84eb.b786 ARPA Vlan52
Internet 172.29.52.106 191 9094.e433.f2c3 ARPA Vlan52
Internet 172.29.52.132 40 e0db.55d9.740d ARPA Vlan52
Internet 172.29.52.147 175 0026.756f.9317 ARPA Vlan52
Internet 172.29.52.171 0 98fc.11e8.3c9e ARPA Vlan52
Internet 172.29.52.237 0 f04d.a266.f0a6 ARPA Vlan52

razam

Internet 172.29.53.1 - 6c20.56cd.7eff ARPA Vlan53
Internet 172.29.53.10 22 0026.756b.25da ARPA Vlan53
Internet 172.29.53.11 47 0016.d48a.9201 ARPA Vlan53
Internet 172.29.53.25 0 bc85.5633.e2bd ARPA Vlan53
Internet 172.29.53.33 180 bcf6.85bf.70d3 ARPA Vlan53
Internet 172.29.53.59 195 00ff.7037.9d0e ARPA Vlan53
Internet 172.29.53.61 0 e89d.872c.901d ARPA Vlan53
Internet 172.29.53.110 21 e8e0.b761.8664 ARPA Vlan53
Internet 172.29.53.115 0 b876.3f27.fc4f ARPA Vlan53
Internet 172.29.53.148 25 e063.e584.0ea6 ARPA Vlan53
Internet 172.29.53.156 133 d422.3f2c.97b9 ARPA Vlan53
Internet 172.29.53.163 94 9018.7cec.3a03 ARPA Vlan53
Internet 172.29.53.164 16 0c14.20d7.fedb ARPA Vlan53
Internet 172.29.53.166 32 0090.a9ce.15e5 ARPA Vlan53
Internet 172.29.53.186 23 c83a.3510.9428 ARPA Vlan53
Internet 172.29.53.188 251 0415.52e9.53e8 ARPA Vlan53
Internet 172.29.53.189 5 a0f3.c189.81b7 ARPA Vlan53
Internet 172.29.53.190 168 7845.c4ae.fc37 ARPA Vlan53
Internet 172.29.53.192 250 8853.2e56.617f ARPA Vlan53
Internet 172.29.53.193 34 e8e0.b718.675b ARPA Vlan53
Internet 172.29.53.195 23 c83a.352c.99e0 ARPA Vlan53
Internet 172.29.53.196 66 206a.8aee.602f ARPA Vlan53
Internet 172.29.53.198 66 94d7.7106.d46f ARPA Vlan53
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.29.53.201 5 a0f3.c178.df5d ARPA Vlan53
Internet 172.29.53.202 0 e89d.87a4.5930 ARPA Vlan53
Internet 172.29.53.203 13 a0f3.c1eb.9d82 ARPA Vlan53
Internet 172.29.53.204 239 782b.cbd5.9090 ARPA Vlan53
Internet 172.29.53.208 3 b4b5.2f7b.5fe0 ARPA Vlan53
Internet 172.29.53.209 1 90f6.5224.9971 ARPA Vlan53
Internet 172.29.53.210 1 687f.7477.6f48 ARPA Vlan53
Internet 172.29.53.215 0 40cb.a82e.0c0c ARPA Vlan53
Internet 172.29.53.216 1 d49a.20ea.4438 ARPA Vlan53
Internet 172.29.53.217 7 a0f3.c1db.1143 ARPA Vlan53
Internet 172.29.53.218 72 98fc.11c2.81b1 ARPA Vlan53
Internet 172.29.53.233 3 f8d1.1196.af07 ARPA Vlan53
Internet 172.29.53.234 239 905f.2e4e.859a ARPA Vlan53
Internet 172.29.53.235 166 a417.313d.5463 ARPA Vlan53
Internet 172.29.53.238 53 6431.508c.43ec ARPA Vlan53
Internet 172.29.53.239 136 e89d.87da.8d2f ARPA Vlan53
Internet 172.29.53.240 144 e063.e565.cc0f ARPA Vlan53
Internet 172.29.53.241 151 f8d1.1188.ba2d ARPA Vlan53
Internet 172.29.53.242 3 f8d1.1188.ba2d ARPA Vlan53
Internet 172.29.53.243 1 88cb.87da.a51f ARPA Vlan53
Internet 172.29.53.244 0 b8e8.5672.8a44 ARPA Vlan53
Internet 172.29.53.245 43 6021.c00c.f82d ARPA Vlan53
Internet 172.29.53.246 0 50cc.f893.0123 ARPA Vlan53
Internet 172.29.53.247 1 0c74.c203.526a ARPA Vlan53
Internet 172.29.54.1 - 6c20.56cd.7eff ARPA Vlan54
Internet 172.29.54.6 153 0025.6447.297e ARPA Vlan54
Internet 172.29.54.7 76 30f9.edb7.3b5d ARPA Vlan54
Internet 172.29.54.31 15 0024.b21a.b7b5 ARPA Vlan54
Internet 172.29.54.78 68 848f.69af.4f30 ARPA Vlan54
Internet 172.29.54.81 259 4c72.b96b.67a4 ARPA Vlan54
Internet 172.29.54.92 31 f4f1.5a98.e0ec ARPA Vlan54
Internet 172.29.54.112 143 5cf8.a100.6ab5 ARPA Vlan54
Internet 172.29.54.113 1 90f6.526d.5f8d ARPA Vlan54
Internet 172.29.54.129 1 a0f3.c1ec.fdc7 ARPA Vlan54
Internet 172.29.54.130 2 f4ec.38d4.7e27 ARPA Vlan54
Internet 172.29.54.137 4 6470.02fc.5b87 ARPA Vlan54
Internet 172.29.54.138 0 90f6.525f.fb95 ARPA Vlan54
Internet 172.29.54.139 4 6470.024c.bf35 ARPA Vlan54
Internet 172.29.54.149 0 6470.02f0.07cb ARPA Vlan54
Internet 172.29.54.150 0 7845.c4a0.d409 ARPA Vlan54
Internet 172.29.54.151 186 001e.33a2.54da ARPA Vlan54
Internet 172.29.54.152 129 8853.2e8d.d781 ARPA Vlan54
Internet 172.29.54.153 0 a0f3.c1db.0961 ARPA Vlan54
Protocol Address Age (min) Hardware Addr Type Interface
Internet 172.29.54.156 0 a45d.3668.c6c0 ARPA Vlan54
Internet 172.29.54.157 52 78e7.d1dc.7438 ARPA Vlan54
Internet 172.29.54.158 128 7486.7a09.bbd6 ARPA Vlan54
Internet 172.29.54.159 4 f8d1.1172.3333 ARPA Vlan54
Internet 172.29.54.160 147 b8ac.6f51.0042 ARPA Vlan54
Internet 172.29.54.161 3 f8d1.11a9.76cd ARPA Vlan54
Internet 172.29.54.165 4 0025.d370.19f5 ARPA Vlan54
Internet 172.29.54.166 0 90e6.ba19.b28d ARPA Vlan54
Internet 172.29.54.167 0 0024.2c27.5ee9 ARPA Vlan54
Internet 172.29.54.168 5 e8e0.b7d5.665b ARPA Vlan54
Internet 172.29.54.173 1 f8d1.117a.6d57 ARPA Vlan54
Internet 172.29.54.183 116 e8e0.b759.675b ARPA Vlan54
Internet 172.29.54.185 223 8853.2e25.61fb ARPA Vlan54
Internet 172.29.54.186 185 3cd0.f86f.43b5 ARPA Vlan54
Internet 172.29.54.187 140 a0f4.19dc.d176 ARPA Vlan54
Internet 172.29.54.188 0 4ceb.4215.22be ARPA Vlan54
Internet 172.29.54.189 88 dc0e.a1ef.49b8 ARPA Vlan54
Internet 172.29.54.190 0 001e.336f.3306 ARPA Vlan54
Internet 172.29.54.198 108 1803.7390.b135 ARPA Vlan54
Internet 172.29.54.204 151 e803.9a0b.f115 ARPA Vlan54
Internet 172.29.54.212 0 c83a.3512.5245 ARPA Vlan54
Internet 172.29.54.235 0 0026.b91b.6c33 ARPA Vlan54

razam

it is too long to paste the full output, have shared partial output of show ip arp

Monkerz

What is 172.29.201.1 within your network? And is a lot of traffic directed to it?

Dieg0M

Change ip route 172.29.201.1 255.255.255.255 GigabitEthernet2/34 => to the IP address of next hop instead and check your CPU utilization.

Monkerz

Dieg0M wrote: »

Change ip route 172.29.201.1 255.255.255.255 GigabitEthernet2/34 => to the IP address of next hop instead and check your CPU utilization.

I was trying to help to OP understand why this is happening, not just tell him how to correct it.

razam

@Monkerz

172.29.201.1 is the WAN ip address. All the traffic is directed to it.

172.29.201.2/30 assigned on Gig2/34 of my core switch (one with high utilization)
172.29.201.1/30 assigned on the neighbor device Gig Interface

@Deig0m,
will remove ip route 172.29.201.1 255.255.255.255 GigabitEthernet2/34 ,
monitor the utilization and share the results.

EdTheLad

In this case that route pointing to Gi2/34 should have no affect as the 172.29.201.0/30 is a directly connected subnet.
It's possible it might be causing an issue for some unknown reason as it's a strange config, it's not required as any traffic destined for .1 will inherently know to go out gi2/34. Remove it see what happens i guess, in your arp outputs alot of addresses have 0 minutes against them, maybe looking at those ip addresses might point towards a common point of failure?

Danielh22185

razam wrote: »

@Monkerz

172.29.201.1 is the WAN ip address. All the traffic is directed to it.

172.29.201.2/30 assigned on Gig2/34 of my core switch (one with high utilization)
172.29.201.1/30 assigned on the neighbor device Gig Interface

@Deig0m,
will remove ip route 172.29.201.1 255.255.255.255 GigabitEthernet2/34 ,
monitor the utilization and share the results.

I am just curious why this would cause / or could cause high CPU utilization and how the difference in changing the IP route to the IP of the neighboring device as apposed to the actual physical IP of the connecting interface.

Dieg0M

Ok, so when you put a static route to a neighboring interface, the router sends an ARP request to find the MAC address of the destination network to forward packets, whether the destination is valid or not. Also, the router will receive an ARP response if another router on the broadcast network is responding on behalf of that network (Proxy ARP). This will cause excessive broadcast traffic on the segment. How I can interpret this configuration is that he is sending default traffic to his next hop 172.29.201.1 but in his local routing table he also has a specific route to this address. Because of this, the router will do a recursive lookup of 172.29.201.1 and do ARP requests for all unknown traffic. The best way to configure this would be:
ip route 0.0.0.0 0.0.0.0 GigabitEthernet2/34 172.29.201.1

This way he will avoid recursive lookups and unnecessary ARP requests. Now in my suggestion I overlooked that his next hop of the static route was infact the same as his default route.

razam

core-switch#show ip arp inspection statistics


Source Mac Validation : Disabled
Destination Mac Validation : Disabled
IP Address Validation : Disabled
No active or enabled vlans on switch.


should these be enabled ?

what is the recommended arp timeout value ?

Dieg0M

Not unless you have Dynamic Arp Inspection enabled to prevent MAC spoofing. Default is 300 sec and I wouldn't change it unless required to.

razam

please see below the pic, high utilization showing because of another process Cat4k Mgmt LoPri

Dieg0M

Give us the output of : sh plat heal

razam

"show platform health" output modified

Dieg0M

Ok check what packets are causing that high cpu: show platform cpu packet statistics

razam

please see the output below for show platform cpu packet statistics

Dieg0M

Did you change the default route to ip route 0.0.0.0 0.0.0.0 GigabitEthernet2/34 172.29.201.1 ? This all seems to indicate excessive ARP lookups.

razam

hi,

yesterday monitored the packets received by the Core Switch,

debug platform packet all receive buffer


show platform cpu packet buffered

saw many broadcast packets by two users, traced the ports of those two users and shut down those ports, since then the CPU Utilization is 10%.

have one question here, i have applied storm control configuration on the access switches, access switch should have put those two ports in an err-disabled state itself.

please see below my interface configuration on access switches.

int range fastEthernet 0/1 - 24/48
description ##TO-END-USERS##
switchport mode access
switchport access vlan 54
speed auto
duplex auto
spanning-tree portfast
spanning-tree bpdufilter enable
no shut
no ip dhcp snooping trust
ip dhcp snooping limit rate 70
storm-control broadcast level 30.00 10.00
storm-control action shutdown
exit


errdisable recovery cause link-flap
errdisable recovery interval 30


it should have put the ports with broadcast storm in an err-disabled state.

EdTheLad

You have the storm control configured for a percentage of bandwidth i.e. 30% for broadcast traffic. So that's allowing 30Mbps of broadcast traffic before the port is shutdown. Lets say the broadcasts are arp requests 46 byte packets.
30,000,000/(46 * = 81521 packets per second.

That would melt your switch, i'm not sure if your switch supports per packet configuration, but if it does its best practices to use pps values. Bandwidth is never the issue when it comes to the control plane, it's always the number of packets to process that's the problem.

razam

my access switch supports pps, please suggest a value for this.

EdTheLad

I cant give you a value off the top of my head, i'd have to look at our network and see what traffic is going where, look at stats etc.
Have a look at the below link for information. Lets say your traffic is 98% unicast and the only broadcast traffic is for arp, then you will have to workout how many end devices are sending arp's to your gateway switch. Looking the your arp cache will give you an idea, remember not all devices send arp's together and this is a packet per second value.
Internetwork Design Guide -- Broadcasts in Switched LAN Internetworks - DocWiki

razam

solved this issue few days back, now its at normal utilization, controlled the broadcast traffic...

the time the utilization went high, monitored those pkts, and chked the mac addresses, then after getting to know about the interfaces of access switches where it was coming, saw the input rate n output rate of those interfaces "show interface fa 0/x",

from the input rate got to know that what value to set for storm control broadcast level.. it has been set at 2.00 rising threshold


thanks everyone for your help