Real World Scenario for everyone ...

jsb515

So at 2am this morning I get a call that one of our offices network went down. So after getting there and noticed that a stack of 4 3750x switches connected by stack cables all had orange lights on them expect the layer 3 uplink ports. Then after consoling into the switches I noticed no vlan database was not present! and VTP was set to Server with domain null but VTP is a layer 2 protocol and the uplinks are set to layer 3. So I had to manually put the vlan info back in from a backup config and got them back up and working.

Anyone have any clue what could cause the vlan database to go proof on a set of switches that have been running fine for almost 4 months now?

cpartin

I think if the domain name is blank the switch will automatically update if it receives any advertisement from another server switch with either a domain name set or a higher vlan database revision number. Put the switches in transparent mode or configure a VTP domain name / password to make sure this can't happen again.

DoubleNNs

Maybe an employee plugged another device into the network which sent VTP messages and wiped the VLAN database out?

TheNewITGuy

show vtp status will give you the last update and from who - do you have logging turned on or config archive as well?

DirtySouth

I can't explain why this happened, but I know from experience that VTP can burn you bad. My recommendation is to do "vtp mode transparent" on ALL Cisco switches in your network. It's really not that difficult to configure VLANs manually and the risk of miss-configured VTP is not worth it.

Iristheangel

As others said, probably someone plugging in another switch in server mode with a higher revision number that wiped everything out.

For smaller sites, I'll use VTP but I always set a VTP domain, VTP version 2, and password and only have one server per site. If I have the option of building a new site from scratch, I'll have a fiber L3 switch (usually 3750-X) and all the access switches home run back to it. Have the 3750 as the VTP master and STP root and all the access layer switches as VTP clients. That way when I have to add another VLAN for a project or whatever, I just log into one switch and BLAM. Propagated to all the other switches on that LAN. When connecting a new switch to the environment, make sure the switch is initially in VTP transparent mode and then change it over to client. Make sure your admins and engineers understand that standard.

For bigger more complex sites or data centers, VTP transparent all the way...

I suppose it's preference but there are some good best practices to have because you never want your VLANs wiped out by one erroneous network addition so put some safeguards in place

jsb515

But we are doing layer 3 no trunking so how can vtp work? Its a layer 2 protocol correct? It said it updated from itself.

Iristheangel

Random question: This is for an office location, right? Are these 4 stacked switches providing access to a small office or something and they function as the access switches as well as L3 for people at that office? The only reason I ask is if the ports aren't hard set to access ports with nonegiotate, those ports could still process DTP messages if someone decided to randomly plug a switch into their desk port. Plug a switch in that's in VTP Server mode (or even VTP client mode with a higher revision number) and only VLAN 1 on it = Buh-bye VLAN database!

Same advice everyone gives you still applies but I would now add to hardcode any non-trunk ports as access ports with nonegotiate.

routergods

As NewITGuy said.. I hope you had config archive set, because there is a really good chance that someone typed in stuff they shouldn't have.

Also, look at the log for that time and see if any ports flapped (which might show cables being plugged in/out).

Friends don't let friends use VTP

jsb515

we don't use vlan 1, but vlan 100 for data and vlan 400 for phone and vlan 300 for wireless. The fiber modules are set to use port channel to the vss core that is one site. The core then uses MetroE from local provider to home back to our main office computer room that also has the same local provider switch that then goes to another switch that is ours that then goes to our distribution vss that then goes to our main core vss...

I always set transparent on switches I setup and ports are always access ports..but I will check and see if auto nonegiotate is on. Unfortunately I did not set these switches up and I even pointed out to them that they have server on. They said they set domain to null and it shouldn't matter....

doing sh log shows nothing unusual any other commands I might not be thinking of?

one of the ports and all 47 others are set the same.

interface GigabitEthernet1/0/1
switchport access vlan 100
switchport voice vlan 400
srr-queue bandwidth share 1 30 35 5
priority-queue out
mls qos trust device cisco-phone
spanning-tree portfast
service-policy input Per-Port-Policing
end

Iristheangel

I wasn't implying you use VLAN 1.

Looks like your port isn't configured exclusively as an access port (switchport mode access). By default, a Cisco switchport is set to dynamic desirable which means if you connect another switch that supports it, it'll negotiate a trunk which could cause that change in VTP/VLANs that you saw there.

Check it out: https://supportforums.cisco.com/thread/147893

RouteMyPacket

Check your logs, check vtp status

If you had ACS in the environment, you could easily discern the idiot who most likely plugged in a switch with vtp "server" with a higher rev number that held no vlan db.

Anyway, good times...get some traceability in your environment man, I wouldn't stand for that situation for a second. If anyone touches my gear, I want to know be it they tell me or I check logs.

Unrelated but I would suggest you configure spanning-tree bpduguard on your access ports as well. Not to mention define them as such with switchport mode access.