Input Errors = Ignored

We are having intermittent errors with one of of 2851 routers. I am seeing bursts of input errors that equal the ignored packets. I noticed the CPU utilization is also high. I am also getting unknown protocol drops.

I have been researching why the input errors = the ignored packets and just wondering if anyone has run into this issue before

Find more posts tagged with

Comments

flipmad

I did some more research and I am seeing alot of throttles. I did a sh interface switching command and the throttle count is very high. The switch connected to this router shows no errors but a high amount of broadcasts.

Its obvious that this router cant handle the traffic it is transferring. im just trying to figure out if it is OSPF, inbound traffic, bad CPU that is causing it.

The buffers show about 13Mbps in and 11Mbps out on both the switch and the router. The GW device is another 2851 with a DS3 connected.

Let me correct myself, this is a 2851 router.

flipmad

I noticed CEF is not enabled on this router. Honestly, I think the root cause of this problem is all the routing decisions it is making. Would enabling CEF benefit me any? Im grasping at straws, but I am trying to figure out a way to prevent the router from making soo many routing decisions and ease up on the processor

This router has about 200-300 neighbors.

networker050184

I'd definitely enable CEF if its not already. This will take some load off of the CPU.

chrisone

yeah also do a

show processes cpu history

and display it for us.

also a

show processes cpu sorted

ipSpace

networker050184 wrote: »

I'd definitely enable CEF if its not already. This will take some load off of the CPU.

I have the same opinion. You are now doing fast switching, so that will use a fair % of the CPU.

Heero

You pretty much ALWAYS want CEF enabled. It is the most efficient routing/switching method that can be used by a typical Cisco device.

flipmad

sh proc cpu hist

07:30:25 PM Thursday Apr 7 2011 UTC

666666777775555566666333335555544444666665555555555666666555
222222222223333300000777773333333333000008888811111000000999
100
90
80
70 *****
60 *********** ***** ********** *********
50 ********************* ***** ************************
40 ************************************************************
30 ************************************************************
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)

765478768645469745869885555555665566678966877786968767866566
256873954883299182583221410888174131106885542188924243064614
100 * * *
90 * * * ** * * *
80 *** * * * *** ** * * * * *
70 ** ****** *#* **##* * **#**##****# *# *#*
60 *** *##*#* *#* #*##* ***** ****######*####*#####*****
50 ##***##*##** *#***#####*#*****##**##########################
40 ###*########*###**########*#################################
30 ############################################################
20 ############################################################
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

999999999899999789899999999999999779999998899999999999998476766677799999
999999998807689708897799999999999489527494909899999999997906173451099749
100 ********* **** * ************** ** * * ************ *** *
90 *************** **************** ****** *************** *****
80 ********************************* *********************** * *****
70 ********************************************************* **** ********
60 ****************************#********************#******* **************
50 #######*****************########***************########****************#
40 ########******########*#################################**************##
30 ########################################################***********#####
20 ############################################################*###*#######
10 ########################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

# sh proc cpu sorted

CPU utilization for five seconds: 37%/21%; one minute: 52%; five minutes: 48%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
204 39441438001607644314 2453 12.01% 22.44% 19.87% 0 OSPF-1 Router
78 21571335961623769588 1328 1.92% 3.14% 2.90% 0 IP Input
33 240 173 1387 0.48% 0.22% 0.05% 322 SSH Process
3 1836369122042849179 89 0.32% 0.85% 0.88% 0 OSPF-1 Hello
132 5516512 196181739 28 0.16% 0.14% 0.16% 0 RBSCP Background
104 812 20006 40 0.08% 0.00% 0.00% 0 TCP Timer
25 24897108 4345771 5729 0.08% 0.11% 0.10% 0 HC Counter Timer
17 56603716 127179888 445 0.08% 0.16% 0.16% 0 ARP Input
211 32596196 4470289 7291 0.08% 0.14% 0.16% 0 Compute load avg
9 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager
10 125380 21479160 5 0.00% 0.00% 0.00% 0 IPC Periodic Tim
8 1572 363803 4 0.00% 0.00% 0.00% 0 IPC Dynamic Cach
13 0 1 0 0.00% 0.00% 0.00% 0 IPC BackPressure
7 0 2 0 0.00% 0.00% 0.00% 0 Timers
11 105548 21479168 4 0.00% 0.00% 0.00% 0 IPC Deferred Por
12 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager
14 0 1 0 0.00% 0.00% 0.00% 0 OIR Handler
18 0 2 0 0.00% 0.00% 0.00% 0 ATM Idle Timer
19 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit
20 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
15 0 1 0 0.00% 0.00% 0.00% 0 Crash writer
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
22 0 2 0 0.00% 0.00% 0.00% 0 DDR Timers
16 157024 727373 215 0.00% 0.00% 0.00% 0 Environmental mo
24 633908 6539469 96 0.00% 0.00% 0.00% 0 EEM ED Syslog
21 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
26 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
27 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers
28 0 2 0 0.00% 0.00% 0.00% 0 SMART
29 150564 21834674 6 0.00% 0.00% 0.00% 0 GraphIt
30 0 2 0 0.00% 0.00% 0.00% 0 Dialer event
31 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
32 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client
6 7542468 13972825 539 0.00% 0.03% 0.02% 0 Pool Manager
34 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest
35 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
23 0 2 0 0.00% 0.00% 0.00% 0 Entity MIB API
37 0 2 0 0.00% 0.00% 0.00% 0 IDB Work
38 189344 7409442 25 0.00% 0.00% 0.00% 0 Logger
36 7727984 16285130 474 0.00% 0.07% 0.07% 0 Net Background
40 839732 21871351 38 0.00% 0.01% 0.00% 0 Per-Second Jobs
41 8004 1453797 5 0.00% 0.00% 0.00% 0 c2800 Periodic
42 0 1 0 0.00% 0.00% 0.00% 0 AggMgr Process
43 0 1 0 0.00% 0.00% 0.00% 0 Token Daemon
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
44 1016 57532 17 0.00% 0.00% 0.00% 0 Transport Port A
45 0 1 0 0.00% 0.00% 0.00% 0 dev_device_inser
46 0 1 0 0.00% 0.00% 0.00% 0 dev_device_remov
47 0 1 0 0.00% 0.00% 0.00% 0 sal_dpc_process
48 0 1 0 0.00% 0.00% 0.00% 0 ARL Table Manage
49 0 2 0 0.00% 0.00% 0.00% 0 ESWPPM
5 36299912 2618056 13865 0.00% 0.12% 0.11% 0 Check heaps
51 0 2 0 0.00% 0.00% 0.00% 0 ESWILPPM
4 0 1 0 0.00% 0.00% 0.00% 0 EDDRI_MAIN
53 692008 86754269 7 0.00% 0.00% 0.00% 0 Netclock Backgro
54 0 2 0 0.00% 0.00% 0.00% 0 SM Monitor
2 680600 4366938 155 0.00% 0.02% 0.00% 0 Load Meter
56 23556 3628385 6 0.00% 0.00% 0.00% 0 mxt5100
57 0 2 0 0.00% 0.00% 0.00% 0 VNM DSPRM MAIN
58 0 1 0 0.00% 0.00% 0.00% 0 DSPFARM DSP READ
59 0 2 0 0.00% 0.00% 0.00% 0 FLEX DNLD MAIN
60 0 1 0 0.00% 0.00% 0.00% 0 HDV background
61 76276 21479119 3 0.00% 0.00% 0.00% 0 Ether-Switch RBC
62 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Pr
63 0 1 0 0.00% 0.00% 0.00% 0 IGMP Snooping Re
64 3076 727376 4 0.00% 0.00% 0.00% 0 Call Management
39 378328 21478985 17 0.00% 0.00% 0.00% 0 TTY Background
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
66 0 1 0 0.00% 0.00% 0.00% 0 CES Line Conditi
67 0 2 0 0.00% 0.00% 0.00% 0 dot1x
68 0 2 0 0.00% 0.00% 0.00% 0 DTP Protocol
69 84992 21479149 3 0.00% 0.00% 0.00% 0 PI MATM Aging Pr
70 11808 2179614 5 0.00% 0.00% 0.00% 0 EtherChnl
71 0 2 0 0.00% 0.00% 0.00% 0 Dot11 auth Dot1x
72 0 1 0 0.00% 0.00% 0.00% 0 Dot11 Mac Auth
73 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
74 72 676 106 0.00% 0.00% 0.00% 0 AAA Server
75 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc
76 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr
77 169380 1243743 136 0.00% 0.00% 0.00% 0 CDP Protocol
50 0 2 0 0.00% 0.00% 0.00% 0 Eswilp Storm Con
79 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl
80 4 4 1000 0.00% 0.00% 0.00% 0 TurboACL
81 0 2 0 0.00% 0.00% 0.00% 0 TurboACL chunk
82 3388 36329 93 0.00% 0.00% 0.00% 0 MOP Protocols
83 0 3 0 0.00% 0.00% 0.00% 0 PPP Hooks
85 0 1 0 0.00% 0.00% 0.00% 0 SSS Manager
86 22536 2907589 7 0.00% 0.00% 0.00% 0 SSS Test Client
87 0 1 0 0.00% 0.00% 0.00% 0 SSS Feature Mana
88 730224 84823859 8 0.00% 0.01% 0.00% 0 SSS Feature Time
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
89 0 1 0 0.00% 0.00% 0.00% 0 VPDN call manage
90 0 1 0 0.00% 0.00% 0.00% 0 L2X Socket proce
91 0 1 0 0.00% 0.00% 0.00% 0 L2X SSS manager
92 0 2 0 0.00% 0.00% 0.00% 0 L2TP mgmt daemon
93 0 1 0 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
94 0 2 0 0.00% 0.00% 0.00% 0 EAPoUDP Process
95 0 2 0 0.00% 0.00% 0.00% 0 IP Host Track Pr
96 0 1 0 0.00% 0.00% 0.00% 0 IPv6 RIB Redistr
97 0 2 0 0.00% 0.00% 0.00% 0 KRB5 AAA
98 190344 518898 366 0.00% 0.00% 0.00% 0 IP Background
99 114580 1106132 103 0.00% 0.00% 0.00% 0 IP RIB Update
100 0 2 0 0.00% 0.00% 0.00% 0 PPP IP Route
1 32 276 115 0.00% 0.00% 0.00% 0 Chunk Manager
102 0 1 0 0.00% 0.00% 0.00% 0 Asy FS Helper
103 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
52 0 2 0 0.00% 0.00% 0.00% 0 Eswilp Storm Con
55 244 4 61000 0.00% 0.00% 0.00% 0 USB Startup
106 0 1 0 0.00% 0.00% 0.00% 0 COPS
108 0 2 0 0.00% 0.00% 0.00% 0 L2MM
109 0 1 0 0.00% 0.00% 0.00% 0 MRD
110 0 1 0 0.00% 0.00% 0.00% 0 IGMPSN
65 0 1 0 0.00% 0.00% 0.00% 0 AAL2CPS TIMER_CU
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
112 0 2 0 0.00% 0.00% 0.00% 0 RLM groups Proce
113 4 2 2000 0.00% 0.00% 0.00% 0 DDP
114 0 2 0 0.00% 0.00% 0.00% 0 SNMP Timers
115 0 2 0 0.00% 0.00% 0.00% 0 SCTP Main Proces
116 0 1 0 0.00% 0.00% 0.00% 0 IUA Main Process
117 246136 21488814 11 0.00% 0.00% 0.00% 0 RUDPV1 Main Proc
118 0 1 0 0.00% 0.00% 0.00% 0 bsm_timers
119 77192 21479152 3 0.00% 0.00% 0.00% 0 bsm_xmt_proc
120 0 1 0 0.00% 0.00% 0.00% 0 CES Client SVC R
121 0 2 0 0.00% 0.00% 0.00% 0 Dialer Forwarder
122 1893148 363373 5209 0.00% 0.00% 0.00% 0 IP Cache Ager
123 34508 365223 94 0.00% 0.00% 0.00% 0 Adj Manager
124 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM Input
125 0 2 0 0.00% 0.00% 0.00% 0 ATM OAM TIMER
126 832 72820 11 0.00% 0.00% 0.00% 0 HTTP CORE
127 8084 168971 47 0.00% 0.00% 0.00% 0 RARP Input
128 0 1 0 0.00% 0.00% 0.00% 0 PAD InCall
129 0 2 0 0.00% 0.00% 0.00% 0 X.25 Background
130 0 2 0 0.00% 0.00% 0.00% 0 PPP Bind
131 0 2 0 0.00% 0.00% 0.00% 0 PPP SSS
101 0 2 0 0.00% 0.00% 0.00% 0 PPP IPCP
133 85292 4348112 19 0.00% 0.00% 0.00% 0 L2F management d
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
134 0 1 0 0.00% 0.00% 0.00% 0 Inspect Timer
135 4176276 334569 12482 0.00% 0.00% 0.00% 0 crypto sw pk pro
136 684 72778 9 0.00% 0.00% 0.00% 0 Authentication P
137 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B
138 0 1 0 0.00% 0.00% 0.00% 0 IPS Timer
139 0 2 0 0.00% 0.00% 0.00% 0 SDEE Management
140 0 1 0 0.00% 0.00% 0.00% 0 IPv6 Inspect Tim
141 0 2 0 0.00% 0.00% 0.00% 0 URL filter proc
142 0 3 0 0.00% 0.00% 0.00% 0 Crypto HW Proc
143 11872 873389 13 0.00% 0.00% 0.00% 0 CRM_CALL_UPDATE_
144 112 2 56000 0.00% 0.00% 0.00% 0 CCVPM_HDSPRM
145 104376 8031491 12 0.00% 0.00% 0.00% 0 FLEX DSPRM MAIN
146 35816 8031489 4 0.00% 0.00% 0.00% 0 FLEX DSP KEEPALI
147 0 4 0 0.00% 0.00% 0.00% 0 HDA DSPRM MAIN
148 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA
149 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr
150 0 1 0 0.00% 0.00% 0.00% 0 Key chain liveke
151 0 2 0 0.00% 0.00% 0.00% 0 LINE AAA
152 568 4897 115 0.00% 0.00% 0.00% 0 LOCAL AAA
153 0 2 0 0.00% 0.00% 0.00% 0 TPLUS
154 0 2 0 0.00% 0.00% 0.00% 0 VSP_MGR
155 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
156 0 3 0 0.00% 0.00% 0.00% 0 Crypto WUI
157 0 2 0 0.00% 0.00% 0.00% 0 Crypto Support
158 0 1 0 0.00% 0.00% 0.00% 0 CCVPM_HTSP
159 0 2 0 0.00% 0.00% 0.00% 0 VPM_MWI_BACKGROU
160 0 1 0 0.00% 0.00% 0.00% 0 CCVPM_R2
161 132 24261 5 0.00% 0.00% 0.00% 0 FB/KS Log HouseK
162 0 2 0 0.00% 0.00% 0.00% 0 EPHONE MWI BG Pr
163 0 1 0 0.00% 0.00% 0.00% 0 CCSWVOICE
165 0 1 0 0.00% 0.00% 0.00% 0 http client proc
167 4 1 4000 0.00% 0.00% 0.00% 0 QOS_MODULE_MAIN
168 0 1 0 0.00% 0.00% 0.00% 0 RPMS_PROC_MAIN
169 0 1 0 0.00% 0.00% 0.00% 0 VoIP AAA
170 0 1 0 0.00% 0.00% 0.00% 0 crypto engine pr
171 40 4 10000 0.00% 0.00% 0.00% 0 Crypto CA
172 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL
173 0 1 0 0.00% 0.00% 0.00% 0 Crypto SSL
174 0 6 0 0.00% 0.00% 0.00% 0 Crypto ACL
175 0 2 0 0.00% 0.00% 0.00% 0 CRYPTO QoS proce
176 0 1 0 0.00% 0.00% 0.00% 0 Crypto INT
177 0 4 0 0.00% 0.00% 0.00% 0 Crypto IKMP
178 11992 1090690 10 0.00% 0.00% 0.00% 0 IPSEC key engine
179 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
180 0 1 0 0.00% 0.00% 0.00% 0 Crypto PAS Proc
181 0 1 0 0.00% 0.00% 0.00% 0 Crypto Delete Ma
182 0 2 0 0.00% 0.00% 0.00% 0 Key Proc
183 99428 21834678 4 0.00% 0.00% 0.00% 0 Crypto Device Up
184 0 2 0 0.00% 0.00% 0.00% 0 Multi-ISA Event
185 0 1 0 0.00% 0.00% 0.00% 0 Multi-ISA Cleanu
186 0 1 0 0.00% 0.00% 0.00% 0 PM Callback
187 0 1 0 0.00% 0.00% 0.00% 0 DATA Transfer Pr
188 0 1 0 0.00% 0.00% 0.00% 0 DATA Collector
189 0 2 0 0.00% 0.00% 0.00% 0 AAA SEND STOP EV
190 0 3 0 0.00% 0.00% 0.00% 0 EEM ED CLI
191 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Counter
192 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Interface
193 0 3 0 0.00% 0.00% 0.00% 0 EEM ED IOSWD
194 4 2 2000 0.00% 0.00% 0.00% 0 EEM ED Memory-th
195 0 2 0 0.00% 0.00% 0.00% 0 EEM ED None
196 0 2 0 0.00% 0.00% 0.00% 0 EM ED OIR
197 0 2 0 0.00% 0.00% 0.00% 0 EEM ED SNMP
198 5252 364063 14 0.00% 0.00% 0.00% 0 EEM ED Timer
199 160320 4355396 36 0.00% 0.00% 0.00% 0 EEM Server
200 10960 2179613 5 0.00% 0.00% 0.00% 0 RMON Recycle Pro
201 0 2 0 0.00% 0.00% 0.00% 0 RMON Deferred Se
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
202 0 1 0 0.00% 0.00% 0.00% 0 Syslog Traps
203 4 2 2000 0.00% 0.00% 0.00% 0 VLAN Manager
105 8564 57397 149 0.00% 0.00% 0.00% 0 TCP Protocols
205 145164 20332572 7 0.00% 0.00% 0.00% 0 trunk conditioni
206 0 1 0 0.00% 0.00% 0.00% 0 trunk conditioni
207 0 2 0 0.00% 0.00% 0.00% 0 EEM Policy Direc
208 0 1 0 0.00% 0.00% 0.00% 0 Syslog
209 0 1 0 0.00% 0.00% 0.00% 0 VPDN Scal
210 66528 1583454 42 0.00% 0.00% 0.00% 0 Net Input
111 0 1 0 0.00% 0.00% 0.00% 0 IP Traceroute
212 7598016 658337 11541 0.00% 0.02% 0.00% 0 Per-minute Jobs
213 0 1 0 0.00% 0.00% 0.00% 0 tHUB
214 514236 2038443 252 0.00% 0.00% 0.00% 0 IP SNMP
216 1520 12129 125 0.00% 0.00% 0.00% 0 SSH Event handle
217 163908 1017517 161 0.00% 0.00% 0.00% 0 PDU DISPATCHER
218 579500 1017395 569 0.00% 0.00% 0.00% 0 SNMP ENGINE
219 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6
220 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
221 0 1 0 0.00% 0.00% 0.00% 0 SNMP Traps

mikej412

If you use the code tags your output would be readable.

flipmad wrote: »

sh proc cpu hist

07:30:25 PM Thursday Apr 7 2011 UTC


    666666777775555566666333335555544444666665555555555666666555
    222222222223333300000777773333333333000008888811111000000999
100
 90
 80
 70       *****
 60 ***********     *****               **********     *********
 50 *********************     *****     ************************
 40 ************************************************************
 30 ************************************************************
 20 ************************************************************
 10 ************************************************************
   0....5....1....1....2....2....3....3....4....4....5....5....6
             0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)


    765478768645469745869885555555665566678966877786968767866566
    256873954883299182583221410888174131106885542188924243064614
100               *                        *        *
 90               *   * *                 **  *   * *
 80     *** *     *   * ***               **  *   * * *   *
 70 **  ******   *#*  **##*        *     **#**##****# *# *#*
 60 *** *##*#*   *#*  #*##*    *****  ****######*####*#####*****
 50 ##***##*##** *#***#####*#*****##**##########################
 40 ###*########*###**########*#################################
 30 ############################################################
 20 ############################################################
 10 ############################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6
             0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%


    999999999899999789899999999999999779999998899999999999998476766677799999
    999999998807689708897799999999999489527494909899999999997906173451099749
100 *********  ****  * **************  ** * *   ************           *** *
 90 ***************  ****************  ****** ***************          *****
 80 ********************************* ***********************       *  *****
 70 ********************************************************* ****  ********
 60 ****************************#********************#******* **************
 50 #######*****************########***************########****************#
 40 ########******########*#################################**************##
 30 ########################################################***********#####
 20 ############################################################*###*#######
 10 ########################################################################
   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
             0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

flipmad

Thanks, sorry about that.

Is enabling CEF on a production network service impacting?

chrisone

Yeah that router is doing heavy work! CEF may work to relieve some stress however something is up with OSPF.

Is this router in area 0? how many neighbors does it have? Are you seeing neighbor resets on OSPF, maybe you have a bad cable/interface where you saw those erros?

Those could all be possibilities.

Also if you have many VLANs and networks, just because they are local dont forget they still have to transverse a router/gateway to reach the other VLANs in your local network. So if you sending gigs of data from one host on one VLAN to another host on another VLAN, all that stuff needs to go through the router/gateway. Just a thought....but that ends up being an interface bandwidth issue, not sure why you have high CPU issue. It could still be related but i still see high OSPF on your processess.

flipmad

I enabled CEF on the router and the errors havent completely cleared, but they are muuch less. Does it basically cache the routes? So OSPF doesnt have to work so hard?

This router is in area 0, the 300+ neighbors are in a secondary area

networker050184

What kind of set up do you have going on where a single router has 300 OSPF neighbors? Thats pretty outrageous. Even in large nation wide networks I've never seen a router get close to 100 OSPF neighbors.

Chipsch

I agree with Networker050184. That is a ton of neighbors, not to mention from what it sounds like those neighbors are all in one area outside of area 0. Not a great design spec to have that going on. So from your show proc cpu sorted output OSPF is causing a lot of problems. How many actually routes are being advertised? Are you aggregating addresses as the ABR? If not do you have a lot of links flapping causing updates to be propagated? Also you can look at tuning the Hello/Dead intervals depending on how stable you feel your network is to alleviate a lot of the chitter chatter.
Defenitely looks like you need to take a look at that Routing Topology though.

As for moving to CEF, good move although the IP input process wasn't anything to alarming at the time of your output to indicate a ton of packets being shipped off to the cpu.

Another thing, if that 2851 has 300 neighbors I would look at a possible hardware upgrade, just to be safe.

flipmad

I did not design this network. Only inherited it, it is one of the larger scale OSPF networks I have been involved in and I want to redesign it to work more efficiently. I included a JPG of a basic design.

Basically, there is redundant DS3 connections, each location has 2 tunnels connected back to both. Half of the tunnels have primary tunnel to HQ1 and the other half is HQ2. This is to ensure that if one of them goes out the other picks up the slack. It seems that all of the remote networks (Area 99) is being advertised at the GRE routers.

Network Example.jpg

chrisone

In the simplest terms you need a hardware upgrade at your ABR to have a quick fix to your problem. That is if you are even having a performance problem.

300 neighbors in one area is not a good idea. You should start thinking about creating different geographic areas to help your Area 0 out from being bombarded with LSAs. OSPF is a high process intense protocol by nature, with poor design it can severely bog down your router. With OSPF you need to design with a hierarchical approach. Your ABR's act like hubs so you need to beef them up.

Just think if one out of those 300 remote sites have a connection loss or someone powered off a device, or the electricity went out on one of those sites, OSPF needs to send updates to every single user in OSPF. Seems like you should spend more time about how to redesign your OSPF network, so in the mean time just request bigger routers, replace both, not just one. You will need the hardware regardless, explain to them about the scalability factor and how you are pushing the routers to the max already with little room for any growth.

flipmad

Definitely agree about upgrading the hardware.

With your proposed design. I have included a quick drawing of it. Geographically sounds like a great idea. I can put them all in their own area. But the IPs are not sequential, so would a better idea be to group them by sequential IPs then summarize the routes.

Like If am subnetting a 192.168.0.0 into blocks of /26's should I just group the sequential networks together regardless of the geographical location?

192.168.0.0
192.168.0.64
192.168.0.128
etc

See attached

network example2.jpg

network example3.jpg

networker050184

Either of those will probably help the HQ sites out. Grouping them by address is only going to help the branch sites though as area 0 is still going to have all of the routes regardless. I'd assume you are just sending a default out there anyway?

chrisone

Yes you can do that but then your geographic Areas are no longer Geographically separated but just logically separated. How big are your remote networks? do they have many users behind them? do they hold many LANs and do they have many servers?

Depending on the size you may look at re-IPing the network, since you are redesigning it anyway. It is a daunting task so that is why i ask how big are these remote sites. If they are like POS systems or stores /satellite office with 2 to 5 users then i would try and go the route of re-iping with a new clean subnet. However redoing 300 of anything is a daunting task. You can create logical areas with the subnets like you mentioned, you can just call the Areas/regions by an octect that represents the network.

flipmad

None of the remote sites have servers. They are all machines that require access to the internal server at HQ. All them are allocated /27's but some of them have a secondary /24 DHCP subnets.

I really appreciate being able to bounce these ideas off of you guys. You guys are always very helpful

networker050184 you always seem to have input in every one of my posts.

Sometimes I wish I knew or worked with some of you.

burbankmarc

If that's the case then do you really need to run OSPF on these facilities?

flipmad

burbankmarc wrote: »

If that's the case then do you really need to run OSPF on these facilities?

The whole point is to be able to have a primary and secondary weighted tunnels. This is much more easily managed by having weighted OSPF routes. Even though the remote sites access HQ only, some of the remote sites require access to other sites. It it was done efficiently, then I think dynamic routing would be the best solution.