Fried NVRAM

I have a couple 2650XMs (among other things) and a 2511 that I'll be using as a terminal server. I wired everything up with an octal cable and set about trying to configure access through the 2511, which did not work, I think for config reasons.

I got frustrated messing with the configs, so I switched on of the 2650XMs back to the regular console cable to try something, and now I seem to have NVRAM issues. The 2650s are the only devices having trouble, and they both seem to have the same problems. Whenever I try to build a configuration or copy the running configuration over, I get "NVRAM Verification Failed".

Also, when I do a reload, right after the bootstrap, I see "Warning: monitor nvram area is corrupt ... using default values" and right after the image decompresses, I see "Invalid iomem value in ROM (174). Smart-init will
now run to compute a valid value". I can also see that register is set at "0x101 (will be 0x0 at next reload)", which seems odd to me. Since the NVRAM wont take any changes, I don't seem to be able to alter that.

I could *swear* that when I got these and powered them up to check them out that I didn't see any of these problems. The only thing that I've really done with them since was to hook them up to the terminal server. I'm almost positive that I hooked everything up properly, is there something that I could have done that would have fried them? Other than the NVRAM problem, everything seems to be working fine.

Find more posts tagged with

Save $250 on 2025 certification boot camps from Infosec!

Book now with code EOY2025

Button

Comments

tiersten

Read this. Its on the first page of results if you google for "monitor nvram area is corrupt" :P

j_a_s_o_n

Sorry, should have put that in there too. I did try that and they both show passed. I also tried the write erase.

tiersten

j_a_s_o_n wrote: »

Sorry, should have put that in there too. I did try that and they both show passed. I also tried the write erase.

Ah okay. My bad.

Hmm... Odd that it is affecting both though. Have you upgraded the IOS recently?

j_a_s_o_n

I haven't upgraded them since I got them. Both are:

System image file is "flash:c2600-is-mz.122-12a.bin

I know that there are newer versions, but I don't have one immediately handy. I should be able to get ahold of one next week to try it.

tiersten

j_a_s_o_n wrote: »

I haven't upgraded them since I got them.

Thats fine. I was just wondering because you get odd errors like this if you have the wrong IOS file loaded or its corrupted. Still very strange that its affecting both routers at the same time though.

Can you post a full log of it booting?

Forsaken_GA

edit: whoops, nm, you answered that

j_a_s_o_n

Here's the log from one of them:

router1#reload
Proceed with reload? [confirm]environment write to NVRAM failed

2d22h: %SYS-5-RELOAD: Reload requested by console.environment write to NVRAM failed

System Bootstrap, Version 12.2(8r) [cmong 8r], RELEASE SOFTWARE (fc1)
Copyright (c) 2003 by cisco Systems, Inc.

Warning: monitor nvram area is corrupt ... using default values
C2600 platform with 262144 Kbytes of main memory

program load complete, entry point: 0x80008000, size: 0x9a9cac
Self decompressing the image : ############################################################################################################################################################################################# [OK]

Invalid iomem value in ROM (174). Smart-init will
now run to compute a valid value
iomem percentage may need to be reset manually
Smart Init is enabled
smart init is sizing iomem
ID MEMORY_REQ TYPE
00036E 0X000BA600 C2650XM Single Fast Ethernet
0X000F3BB0 public buffer pools
0X00211000 public particle pools
TOTAL: 0X003BF1B0

If any of the above Memory Requirements are
"UNKNOWN", you may be using an unsupported
configuration or there is a software problem and
system operation may be compromised.
Rounded IOMEM up to: 4Mb.
Using 3 percent iomem. [4Mb/128Mb]
environment write to NVRAM failed

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IS-M), Version 12.2(12a), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Tue 24-Sep-02 03:27 by pwade
Image text-base: 0x8000808C, data-base: 0x810CE560

cisco 2650XM (MPC860P) processor (revision 0x100) with 126976K/4096K bytes of memory.
Processor board ID JAD065004X2 (1282312535)
M860 processor: part number 5, mask 2
Bridging software.
X.25 software, Version 3.0.0.
1 FastEthernet/IEEE 802.3 interface(s)
1 Serial network interface(s)
32K bytes of non-volatile configuration memory.
49152K bytes of processor board System flash (Read/Write)

--- System Configuration Dialog ---

Would you like to enter the initial configuration dialog? [yes/no]:
%Error opening tftp://255.255.255.255/network-confg (Timed out)
%Error opening tftp://255.255.255.255/cisconet.cfg (Timed out)
%Error opening tftp://255.255.255.255/router-confg (Timed out)
%Error opening tftp://255.255.255.255/ciscortr.cfg (Timed out)
%Error opening tftp://255.255.255.255/network-confg (Timed out)
%Error opening tftp://255.255.255.255/cisconet.cfg (Timed out)
%Error opening tftp://255.255.255.255/router-confg (Timed out)
%Error opening tftp://255.255.255.255/ciscortr.cfg (Timed out)
%Error opening tftp://255.255.255.255/network-confg (Timed out)
%Error opening tftp://255.255.255.255/cisconet.cfg (Timed out)
% Please answer 'yes' or 'no'.
Would you like to enter the initial configuration dialog? [yes/no]: no

Press RETenvironment write to NVRAM failed
environment checksum in NVRAM failed
URN to get started!

%Error opening tftp://255.255.255.255/router-confg (Timed out)
00:00:08: %LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
00:00:08: %LINK-3-UPDOWN: Interface Serial0/0, changed state to down
00:00:09: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
00:00:09: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, changed state to down
00:00:19: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
00:00:35: AUTOINSTALL: FastEthernet0/0 is assigned 10.0.0.193
00:05:03: %IP-5-WEBINST_KILL: Terminating DNS process
00:05:05: %LINK-5-CHANGED: Interface Serial0/0, changed state to administratively down
00:05:08: %SYS-5-RESTART: System restarted --
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IS-M), Version 12.2(12a), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Tue 24-Sep-02 03:27 by pwade
00:05:08: %SNMP-5-COLDSTART: SNMP agent on host Router is undergoing a cold start
Router>
Router>

captobvious

%Error opening tftp://255.255.255.255/network-confg (Timed out) message

tiersten

captobvious wrote: »

%Error opening tftp://255.255.255.255/network-confg (Timed out) message

The problem is that his NVRAM appears to be broken in both routers so turning off that option wouldn't be possible.

tiersten

Searching for the error "Invalid iomem value in ROM" only gives a handful of results for pages/forums about Dynamips and 1 eBay auction for a 3640. From what the Dynamips pages suggest, that error occurrs when you have the wrong router type selected.

I'd say try to reload IOS on one of the routers.

The other thing I can think of is that the cookie is screwed up. Do the reported MAC addresses of the internal interface match up? Can you do a "show diag" and "show inv" and paste the results?

Still puzzled as to why its affected both routers at the same time though. I can understand a freak incident where one router went crazy and corrupted the NVRAM but both at the same time is a tad unlikely. Both routers worked okay previously and you've altered the configurations on them before I assume?

j_a_s_o_n

Here's the output from show diag:

Slot 0:
C2650XM 1FE Mainboard Port adapter, 2 ports
Port adapter is analyzed
Port adapter insertion time unknown
EEPROM contents at hardware discovery:
Hardware Revision : 1.0
PCB Serial Number : JAD065004X2 (1282312535)
Part Number : 73-7755-02
RMA History : 00
RMA Number : 0-0-0-0
Board Revision : A0
Deviation Number : 0-0
EEPROM format version 4
EEPROM contents (hex):
0x00: 04 FF 40 03 6E 41 01 00 C1 18 4A 41 44 30 36 35
0x10: 30 30 34 58 32 20 28 31 32 38 32 33 31 32 35 33
0x20: 35 29 82 49 1E 4B 02 04 00 81 00 00 00 00 42 41
0x30: 30 80 00 00 00 00 FF FF FF FF FF FF FF FF FF FF
0x40: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
0x50: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
0x60: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
0x70: FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF

WIC Slot 0:
Serial 1T WAN daughter card
Hardware revision 1.0 Board revision B0
Serial number 32056578 Part number 800-01514-02
Test history 0x0 RMA number 00-00-00
Connector type Wan Module
EEPROM format version 1
EEPROM contents (hex):
0x20: 01 02 01 00 01 E9 25 02 50 05 EA 02 00 00 00 00
0x30: 58 00 00 00 04 03 04 01 FF FF FF FF FF FF FF FF

But show inv appears not to be a valid command on this router. I just got these routers a month or so ago, so this is the first time that I've really tried to do much with them. I powered them up when I first got them and made sure that everything looked like it came up ok, but didn't try configuring them (now I know better for next time). I'm reasonably sure that none of these errors showed up when it was booting previously though. As far as I can tell from looking at the cisco page, c2600-is-mz.122-12a.bin should be valid for these. I'll see if I have anything else around that will work to try loading on them.

j_a_s_o_n

Also here's the output from show interfaces:

Router#show interfaces
FastEthernet0/0 is up, line protocol is up
Hardware is AmdFE, address is 000b.be49.0e20 (bia 000b.be49.0e20)
Internet address is 10.0.0.193/24
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:04, output 00:00:07, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
6445 packets input, 1131673 bytes
Received 6318 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
2876 packets output, 352299 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Serial0/0 is administratively down, line protocol is down
Hardware is PowerQUICC Serial
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation HDLC, loopback not set
Keepalive set (10 sec)
Last input never, output never, output hang never
Last clearing of "show interface" counters 05:45:34
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
DCD=down DSR=down DTR=down RTS=down CTS=down

The mac matches what I can see over the network for the IP

tiersten

j_a_s_o_n wrote: »

But show inv appears not to be a valid command on this router.

Yeah. I just looked it up. show inventory was only added in 12.3.4T and you're running 12.2.

j_a_s_o_n wrote: »

As far as I can tell from looking at the cisco page, c2600-is-mz.122-12a.bin should be valid for these.

From what Cisco has on their site for download still, that looks like a valid image at least from the filename. The 2651XM images on the CCO site all start with c2600 anyway and nothing 2650XM specific. Not 100% sure however since they've removed a lot of the old releases due to bugs or various other issues.

Everything is being detected correctly so it isn't the cookie.

If you've tried the troubleshooting steps in the original page I linked to and you've not changed anything else then I'm unsure what is going on and how to fix it. The best I can offer is to try reloading IOS on it.

j_a_s_o_n

I'll give reloading it a try, but I may not get to it until Monday. Thanks for the help!

j_a_s_o_n

Ok, I had a chance to load a new image on one, but no dice. There is a new error message on this one (bolded below), but I didn't come up with anything on it:

System Bootstrap, Version 12.2(8r) [cmong 8r], RELEASE SOFTWARE (fc1)
Copyright (c) 2003 by cisco Systems, Inc.

Warning: monitor nvram area is corrupt ... using default values
C2600 platform with 262144 Kbytes of main memory

program load complete, entry point: 0x80008000, size: 0xf55678
Self decompressing the image : ######################################################################################################################################################################################################################################################################## [OK]

Invalid iomem value in ROM (174). Smart-init will
now run to compute a valid value
iomem percentage may need to be reset manually
Smart Init is enabled
smart init is sizing iomem
ID MEMORY_REQ TYPE
00036E 0X000B3B80 C2650XM Single Fast Ethernet
0X00098670 public buffer pools
0X00211000 public particle pools
TOTAL: 0X0035D1F0

If any of the above Memory Requirements are
"UNKNOWN", you may be using an unsupported
configuration or there is a software problem and
system operation may be compromised.
Rounded IOMEM up to: 4Mb.
Using 3 percent iomem. [4Mb/128Mb]
environment write to NVRAM failed

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IK9O3S3-M), Version 12.3(24), RELEASE SOFTWARE (fc4)
Technical Support: Cisco - Shortcut
Copyright (c) 1986-2007 by cisco Systems, Inc.
Compiled Thu 18-Oct-07 14:26 by stshen
Image text-base: 0x80008098, data-base: 0x81A1E1E8

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
Export Compliance Product Report Application

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco 2650XM (MPC860P) processor (revision 0x100) with 126976K/4096K bytes of memory.
Processor board ID JAD065004X2 (1282312535)
M860 processor: part number 5, mask 2
Bridging software.
X.25 software, Version 3.0.0.
1 FastEthernet/IEEE 802.3 interface(s)
32K bytes of non-volatile configuration memory.
49152K bytes of processor board System flash (Read/Write)

NV: Invalid Magic(0xA6A6) found in NVRAM.....Erase of configuration files recommended

%Error opening tftp://255.255.255.255/network-confg (Timed out)
%Error opening tftp://255.255.255.255/cisconet.cfg (Timed out)

%Error opening tftp://255.255.255.255/router-confg (Timed out)environment write to NVRAM failed

%Error opening tftp://255.255.environment write to NVRAM failed
environment checksum in NVRAM failed
255.255/ciscortr.cfg (Timed out)

Press RETURN to get started!

*Mar 1 00:00:-5-RESTART: System restarted --
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IK9O3S3-M), Version 12.3(24), RELEASE SOFTWARE (fc4)
Technical Support: Cisco - Shortcut
Copyright (c) 1986-2007 by cisco Systems, Inc.
Compiled Thu 18-Oct-07 14:26 by stshen
*Mar 1 00:01:55.196: %SNMP-5-COLDSTART: SNMP agent on host Router is undergoing a cold start
%Error opening tftp://255.255.255.255/network-confg (Timed out)
*Mar 1 00:02:04.155: %SYS-4-CONFIG_RESOLVE_FAILURE: System config parse from (tftp://255.255.255.255/network-confg) failed
Router>
%Error o

I did, however, find something that seemed a bit odd to me. I decided to crack the cases and check to make sure that everything was seated. I noticed that on both 2650XMs, I had an empty socket where I believe boot roms go. To check, I opened a 2611XM, which has something in both sockets. Am I right in thinking that there should be something in there? Pics below, sorry for the blurriness...

This is the 2611XM

This is 2650XM #1

This is 2650XM#2

mikem2te

I did, however, find something that seemed a bit odd to me. I decided to crack the cases and check to make sure that everything was seated. I noticed that on both 2650XMs, I had an empty socket where I believe boot roms go. To check, I opened a 2611XM, which has something in both sockets. Am I right in thinking that there should be something in there? Pics below, sorry for the blurriness...

I think only the one o them is the bot rom.

Can you make out the part number of the chip that is in the 2611 but not in the 2650?

A google seach may reveal what it actually does.

j_a_s_o_n

After poking around a bit, it looks like the left socket is the boot rom. I haven't seen anything that says what the right one is. The left socket is U24 on all of them and the right is U25. Here are the chips:

2611XM:

In U24 is a white labeled chip marked
17-6824-02
CS=141F
0421 H5

In U25 is a black chip marked
CSI
CAT28C256 N
I-12
00427C

2650XM#1:

In U24 is a black chip marked
Winbond
W27E040P-12
282475901
9390EVK
U25 is empty

2650XM#2:

U24 is a white labeled chip marked
17-68
24-02
CS=141F
0432 HM
U25 is empty

It almost looks like U24 on the 2650XMs are two halves of a set. I'm sorely tempted to put them both onto one board to see what happens, but I'm afraid that I'd let out the magic smoke.

j_a_s_o_n

According to this:

Inside Cisco 2651XM — Man v. Cisco

U24 is the ROMMON and U25 is the NVRAM, at least on a 2651XM. I imagine that missing the NVRAM chip entirely might be contributing to my issues. They came from a dealer well known here but are outside of the standard warranty period. I'm going to ping them anyway and see what they say.

captobvious

CAT28C256 256K-Bit Parallel EEPROM

That definitely looks like your culprit. Strange that someone would harvest this before selling.

j_a_s_o_n

The thing that I find really bizarre is that everything works normally, other than writing a config. I would think that the total lack of being able to talk to the NVRAM chip would throw a bigger error than it does.

mikem2te

j_a_s_o_n wrote: »

After poking around a bit, it looks like the left socket is the boot rom. I haven't seen anything that says what the right one is. The left socket is U24 on all of them and the right is U25. Here are the chips:

2611XM:

In U24 is a white labeled chip marked
17-6824-02
CS=141F
0421 H5

In U25 is a black chip marked
CSI
CAT28C256 N
I-12
00427C

2650XM#1:

In U24 is a black chip marked
Winbond
W27E040P-12
282475901
9390EVK
U25 is empty

2650XM#2:

U24 is a white labeled chip marked
17-68
24-02
CS=141F
0432 HM
U25 is empty

It almost looks like U24 on the 2650XMs are two halves of a set. I'm sorely tempted to put them both onto one board to see what happens, but I'm afraid that I'd let out the magic smoke.

So one of them is the boot rom and the other is the NVRAM chip, good to know.

I'm wondering if it is a matter of just buying one from an electronics supplier and banging it in there. It looks like Farnell has a CAT28C256GI-12 which may looks like it is compatible

tiersten

j_a_s_o_n wrote: »

The thing that I find really bizarre is that everything works normally, other than writing a config. I would think that the total lack of being able to talk to the NVRAM chip would throw a bigger error than it does.

The configuration is loaded in at startup and from then on, its working from the in RAM copy. It can get the config via TFTP as well so a NVRAMless router isn't that useless. Still odd though.

You should test your purchases more thoroughly tho!

j_a_s_o_n

mikem2te wrote: »

I'm wondering if it is a matter of just buying one from an electronics supplier and banging it in there. It looks like Farnell has a CAT28C256GI-12 which may looks like it is compatible

I had the same thought myself. The thing that concerns me is that Cisco says that it's not a field replaceable unit. The other thing to try would be to get a low end 2600 from ebay and pull the chip from it.

j_a_s_o_n

tiersten wrote: »

The configuration is loaded in at startup and from then on, its working from the in RAM copy. It can get the config via TFTP as well so a NVRAMless router isn't that useless. Still odd though.

You should test your purchases more thoroughly tho!

Yup, I actually got one of them to pull a config from my tftp server successfully, so at least they'll work that way if I can't get the chip straightened out.

This was definitely a lesson in equipment testing for me. My excuse is being new to cisco, but I've been around computers long enough to know better. I've got some new equipment coming, so I'll be a bit more thorough this time. I'll be cracking the cases on everything to make sure that all of the bits are in there for sure

tiersten

j_a_s_o_n wrote: »

I had the same thought myself. The thing that concerns me is that Cisco says that it's not a field replaceable unit. The other thing to try would be to get a low end 2600 from ebay and pull the chip from it.

Assuming you really are missing the NVRAM, you should be able to just get any compatible chip. It may be listed as a non FRU item but they say that about a lot of the parts in there. They just don't want goons with a big screwdriver prying out the PLCC chip and breaking the socket or PCB. It is not something you replace because you're never really supposed to need to replace it. If it breaks then its a service call.

j_a_s_o_n

I'm waiting to hear from the dealer that I got them from, hopefully they'll send me a couple. If not, mouser has the CAT28C256GI-12 for $6 each, so that's not too bad.

j_a_s_o_n

Just heard back from the dealer (ciscokits btw), they're going to send me the replacement NVRAM chips later this week.