Horror story

Cucumber

Today was a bad day, I want to share my experience with you guys.

Tonight I tried to replace a 6503 with another 6503, same model. I thought it was going to be an easy task, "what could ever be wrong?" I thought to myself, I set up a TFTP server on my laptop, copied the original configuration on the tftp server, and my plan was to do a regular "copy tftp startup" and get out early enough to have some beers.

Well, it turned out to be hell on earth.

I turned on the new switch, set up a VLAN, configured a SVI, configured a port for my laptop, and I was ready to do the copy tftp startup. It didnt work.
Why I dont know, I could ping the SVI ip address from my laptop, but I couldnt ping the
laptop's IP address from the switch. hahaha no "copy tftp startup" for me.

I had to rely on the old and ugly "copy-paste" of commands.

Everything was good, but then, I checked my vlan databse, no vlans! Why? I asked, oh yeah, VTP issuse. This new switch should import the VLAN database from another 6503 which already was visible using "show cdp neighbors". Well it really never updated the VLAN database even though the VTP domain name was ok, Why? I didnt know. hahaha no vlans for me tonite.

I asked for help via telephone support.
"Reload your switch so the VLAN database gets updated through VTP" was the advise,
I followed it stupidly, I reloaded the switch just to find out the console asked me for a password.
hahaha, no console access for me anymore! Why? "something went wrong with the cut and paste of commands" I thought. hahaha I suck.

What I did next? Well, a regular password recovery procedure, I restarted the switch, pressed the CTRL+BREAK combo, and entered rommon, entered confreg 0x2142, and typed i to reload the switch. This is where I really realized how much time do these switches take to boot, I didnt really take the time but I would say they take about 5 minutes to load. Anyway, after this long wait, I still was asked for a password, Why? I didnt know, hahaha, still no console access for me.

My final troubleshooting step was to reconnect the original 6503, after 3 lost hours.

At this point I think I really suck at cisco equipment

EdTheLad

Looks like you went in under prepared.You were lucky you didnt screw the whole network.Did you check the vtp config revision number? If the vtp synched up you could have lost everything.
Its funny somethimes when you're in front of the equipment all logic goes out he window, that why before making any changes i like some time to understand the environment i'm working in.
For the tftp did you setup the tftp server i.e. "tftp-server flash:config"

dtlokee

This is why Cisco exams are so detail oriented; it's the smallest details that will crash your company's network.

As for the ping problems, I would guess you had a firewall on your laptop.

As EdTheLad said, you could have overwritten the VTP domain on the existing switch if you didn't check the VTP revision number, that would have been interesting if you deleted all the VLANs off the witch.

Better luck next time

Cessation

Sounds like a rough night.
Good information though.

Paul Boz

Want to know a horror story? Last night one of our admins swapped out an OC3 card in a Cisco 10k8 and lowered the DHCP lease time in the DHCP server to five minutes so the PVC's would auto-build themselves when new traffic hit the card. It works fine, except for the fact that he forgot to re-raise the lease time, so around 8:30am our tech support department got slammed with a perpetual 50-60 customers in the queue because 40,000 dynamic customers were requesting DHCP ip addresses every 2.5 minutes. The server couldn't handle so many requests and just started discover/offering everyone. Wee...

As for your issue, re-stage that thing using the TFTP method when it's not in your production network. You can do that and take your sweet time doing so rather than doing it in a service window. It makes a hardware replacement virtually transparent. Also, getting in the habit of doing daily backups of your equipment configs is a great way to save your butt in case something awful (like your VTP domain being screwed up) happens.

ITdude

Ouch! That was a bad night. I hope you went out and had some beers or maybe something even stiffer, anyway.

dtlokee, beat me to the punch. I was going to ask if you had something like Zone Labs or Windows Firewall possibly blocking things there....ipso facto now, though.

Rearden

For this reason, you may want to take advantage of the ability to have cisco gear copy it's configuration to a tftp server on demand via an snmp write.

Paul Boz

Rearden wrote:

For this reason, you may want to take advantage of the ability to have cisco gear copy it's configuration to a tftp server on demand via an snmp write.

It can VERY easily be scripted. Just create a process that automatically telnets to the router and copies the config nightly. That way user error is removed.

Rearden

That's what I mean. . . The first time I did this, I used Expect + Perl to do it. However, doing it this way forces you to put the line + enable passwords into the script itself. Using an SNMP Perl module, you can just have it copy itself, using a few OIDs. Cisco even has instruction on the OIDs needed to do this. However, on CPAN there is a Cisco configuration copying module that automates this for you.

I would never do it manually with snmpset and friends.

Paul Boz

Rearden wrote:

That's what I mean. . . The first time I did this, I used Expect + Perl to do it. However, doing it this way forces you to put the line + enable passwords into the script itself. Using an SNMP Perl module, you can just have it copy itself, using a few OIDs. Cisco even has instruction on the OIDs needed to do this. However, on CPAN there is a Cisco configuration copying module that automates this for you.

I would never do it manually with snmpset and friends.

Got ya, I have the same deal working with python and it works fine.

Cucumber

Thanks for the answers and suggestions guys, yeah I checked the VTP revision number, it was zero, the other switch had a revison number of 113. And regarding the laptop, I used the same laptop to store the configuration via tftp, each time I fire up solariswind the firewall asks me whether to enable the access, I always enable it, thats why it was so weird.

APA

What VTP mode was the new switch running in??? All switches in a VTP domain need to be running the same mode......

I've come across instances where VTP clients will not update the VLAN Database because it is running in the incorrect mode.....Sometimes the mode will change automatically and the VTP information will be pushed through.... Other times I've have to manually change the VTP mode so the VTP information will propagate....

Cucumber

It was in client mode, I think I will give it a second try this monday!