What a supposed CCIE is asking to do my core live network (cisco 6500 VSS)

I wonder what his number is. I've seen a lot of the lower numbers be really out of touch with current technology but they were masters of frame relay and ATM back in the day :P

So your in agreement a trunk link between VSS chassis is not what you would expect a CCIE to suggest.

He is actually supposedly quite a recent CCIE! (last recertified 2011)

Well, really hard to say without knowing what the end goal of this design is. Is there a reason to segregate the traffic on a separate link?

networker050184 wrote: »

Well, really hard to say without knowing what the end goal of this design is. Is there a reason to segregate the traffic on a separate link?

But can you actually do it and get it working. you cant configure the VSL links between a VSS pair, so even if you created a second trunk link I can't see how you would block the traffic on the VSL links?

I mean if you have two separate switches you can create two separate trunks and allow vlan A on one and Vlan B on the other or use STP to make separate the traffic and have fail over.

But if you had a stack of 3 X 3750's stacked with stacking cables, would you ever consider connecting port 24 on the first and last switch with UTP cable, and then saying traffic from port A on switch A and Port A on switch C, will travel across this new link? indeed can you block vlans on the stacking ports so they would use it? Because once you configure a port channel into VSL/VSS links you can no longer put any other configuration on them like permit/blck vlans.

Honestly I'm not sure if you can do it at layer two on a VSS pair. I have seen legit reasons (though more hacks than anything) for doing stuff like this. Again, hard to say without knowing more about the end goal and specifics behind it.

bermovick

Even though they're a single logical switch, the 20G VSS link is still used to get data over the "backplane" between the 2. I can't really see any reason to require segregation at that level, but like networker said, it depends on the design.

I have to wonder if it would even work though unless you do a second VSL - at which point why not just make a 40G bundle? (I only know VSS in theory as we have 4500's here and would take multiple updates to be VSS capable).

I mean - without a VSL you're doing the equivalent of looping 2 ports on the same switch and telling traffic to use that between initial ingress and final egress. I am quite curious now though.

DevilWAH wrote: »

So your in agreement a trunk link between VSS chassis is not what you would expect a CCIE to suggest.

I agree with you. With VSS, both switches are sharing the same control plane. From what I know, tt's effectively the same as taking one switch and plugging a CAT6 into G1/0/1 and G1/0/2. Sounds like this guy is thinking of Nexus and vPC where you don't share the control plane and you can prune the VLANs on the peer link - there's a real design reason for doing that. Maybe he's confused or maybe there is some weird design goal he's looking to do. It seems weird that he'd want to filter VLANs between a VSS pair though.

The backstory of this is that we have 6 ESXi servers (3 in to each chassis) and the are set up to run Virtual SAN (vSAN) but the performance is awful. Now our 20Gig VSL links is never above 20% utilisation and neither ports connected to servers or the VSL show any dropped packets. But the company who set up the ESXi servers want to eliminate the network.

So they want to enable Jumbo frames and do this strange config to segregate the traffic. Talking to other Vmware expertise they have run Vsan over 2 X 1gig links with better performance than what we are seeing. But rather than try to simplify the set up they just keep asking to make all these changes making it more and more complicated so the issue is harder and harder to pinpoint!

So storage traffic is going over TCP/IP on these VSS switches? If jumbo frames weren't enabled earlier, that could have been the poor performance. I saw a customer with a C-series UCS server connected to 2960s connecting down to Netapp appliances and iSCSI was being pushed to Netapp. Someone forgot to configure jumbo frames and performance was BAD even though they were pushing 1Gig links. Turned on jumbo frames and BOOM. Fixed.

There's a lot of ways they can segment the traffic here but based on what you're saying, this sounds like a weird hack or design for this guy to be doing. Granted, I'm not the expert (yet!) and I'm not a VMWare expert by any means nor I don't know your infrastructure but it sounds odd to me.

Does sound like he's not completely understanding the VSS concept. I believe the VSL supports jumbo frames by default anyway so if your edge ports are set up you are good to go.

Can you just hook them all into one chassis for now to rule it out? That would be my first suggestion.

I have seen it where HAving jumbo frames enabled on the servers but not on switch path has caused massive latency issues due to fragmentation. Turning of jumbo on servers or enabling it on the switch path solved the issue, it was mismatch that was the issue not the frame size itself. VMware say there is a 5-10% improvement on performance., and my view was to move the servers (that are in test phase) to a single switch off the network and disable jumbo frame throughout. and once that is working build it back up.

but no they want to make it as complicated as possible in an attempt to get to the cause!

I'd definitely make sure jumbo frames are enabled through the path. I'm not a VM/storage expert by any means, but I've always been told that it is best practice there with any sort of virtual storage.

networker050184 wrote: »

I'd definitely make sure jumbo frames are enabled through the path. I'm not a VM/storage expert by any means, but I've always been told that it is best practice there with any sort of virtual storage.

Best practices are not always good for when trying to troubleshoot and issue. For example when you are setting up a web site application, you might while testing decided to turn of some security so its not another factor to contend with. once you have the application running correctly then you enable the full security and lock it all down.

Same for jumbo frames, yes they give you better performance, but that is really all they give. so if you are expecting a throughput of 500Mbytes/s on a storage system and you are getting 10, the issue is not jumbo frames. So knowing all the issues you get with jumbo frames if they are not implemented correctly my first step would be to disable them through out so I know misconfigured jumbo frames are not the issue and work to get a half decent performance and stable performance and only then re enable jumbo frames to get that last bit of performance. (generally on a SAN jumbo frames will give you up to 20-30% increases and lower CPU utilization)

I kind of take a different look at it personally. Set it up following best practices and see where you are. Then troubleshoot as needed. Not the opposite like you are approaching it.

lsud00d

Has this been occurring since day 1, or did something happen along the way? i.e. was there any gear replacement in the path, end to end?

networker050184 wrote: »

I kind of take a different look at it personally. Set it up following best practices and see where you are. Then troubleshoot as needed. Not the opposite like you are approaching it.

Best practices have been configured and don't work. And the key to good trouble shooting is understanding why best practices are best practices. I know best practices are to enable SSH on a switch for management. But if interface X is not authenticating dot1x clients, I know that changing from telent to SSH is not going to solve the issue.

Same with jumbo frames. if my CPU is low and the link utilization is low, if my latencies are 100ms, switching on jumbo frames are is not going to solve the issue. Piuss I know that if I enable jumbo frames and get it wrong then thats another variable to throw in the mix.

It is a rare situation when you can meet 100% of best practices 100% of the time for 100% of you systems, money does not allow it. or different venders do not permit them to be simultaneously configured. This where you need to go though the best practices (with a subject matter expert if needed) and confirm what is requirements and what the effect of not meeting a best practice will have. read Microsoft best practices and you will often see that there best practice is to use a 3rd party application and more often than not they will say to use their own solution. I have seen microsoft best practice for setting up DHCP to configure it to update windows DNS. Now if you don't have a windows DNS server this does not mean the system will not work. Just you have to know a bit more to integrate it in to your own DNS solutions if that is required.

In regard to this issue, VMware themselves state "jumbo frames are not required and will not improve performance in all case", it is an optional configuration, not a requirement. (indeed there VSAN design and guidance guide does not even mention jumbo frames).

Essendon

@DevilWAH, quick question for ya, are all hosts contributing to the VSAN cluster? Apparently performance's poor if not all ESXi hosts in the cluster participate in the VSAN cluster. Also, from your other thread, have you had a chance to find out the PSP you've got going in ESXi? If it's fixed, changing to Round Robin might just do it because it seems like all IOs that are being sent down one link are smashing it resulting in poor performance.

Another question - is the 10GbE dedicated to VSAN and have you looked at what VSAN observer says about the situation?

And read this please > http://www.yellow-bricks.com/2013/09/09/vmware-vsphere-virtual-san-design-considerations/ and this for some performance related info >
http://www.reddit.com/r/vmware/comments/2799p4/root_cause_analysis_of_my_vsan_outage/

Dieg0M

Every CCIE I know will always go with unconventional practices. I think they miss their CCIE Lab exam days.

phoeneous wrote: »

Ask him to explain to you exactly why he's trying to setup it up like that. Maybe mock it up in a lab? If it were me, I'd be looking at storage first.

+1 - You're right - Storage is a good place to look as well. I'm always thinking from a network perspective first but I've seen some storage issues like you wouldn't believe. It's never fun to watch an undersized Netapp deployment go under because that part of the design is just horrible.

deth1k

Not every CCIE has experience in configuring / troubleshooting VSS, it's like asking a mechanic to fix F1 car. Why not approach the guy and ask what exactly is he doing?

deth1k wrote: »

Not every CCIE has experience in configuring / troubleshooting VSS, it's like asking a mechanic to fix F1 car. Why not approach the guy and ask what exactly is he doing?

I agree, but if I am working on a customers network and I am asking them to make changed to there core network. Then if I have not worked on it before I research what I am asking or get a second opinion.

If I take my F1 car to the garage down the road and ask the engineer to fix an oil pressure issue, I expect them as professionals to hold up there hands and say they have not worked on it before and need to seek a second opinion, or at least look at the basic tecnical specs of the car to know what they are dealing with. I would not expect them to pull of the nose of the car and look puzzled when they cant find the engine.

If you are a decent engineer and you come across a set up that you have not had experience with before the first thing you do, before making any changes is to spend 5 minutes familiarising your self with it.

Its not like he has had 5 minutes with this, he has remote access to the switches to view the configs, and has been working on this for a few days before making the suggestion. So this was not an "on the spot" suggestion he made, it was thought out and prepared in to an email that was sent to all parties involved by this engineer.

We have spoken and I know exactly what the intention of doing it was, and that was a reasonable suggestion, but then blindly suggesting major config changes is unprofessional and not what we are paying for. If I am paying my mechanic to fix my F1 car I either expect him to know what he is doing and do it correctly, or tell me he is not able to. I am paying him because he is the "expert" I don't expect to have to watch over the mechanic to insure he does not break it further.

And it is not like asking a mechanic to fix a F1 car, its like asking a ford car mechanic to fix a Suzki car and him struggling to change the tyres. you might need different equipment for the two tasks but the basic concept is the same. The basic fundamentals of VSS + the basic fundamentals of switched networking and his suggestion is flawed. Espicaly considering when he was asked how it would be implemented he admitted he did not know "the commands". Again if you want to suggest changes to a network then before suggesting them its a good idea to form a basic implementation plan.

joelsfood

Absolutely right, deth1k. I've worked on over 100 6500s, including multiple side by side pairs, and never touched VSS (not that I'm a CCIE yet

).

While jumbo frames are helpful, if cpu is low on the sups, and there aren't errors, etc, I'm hesitant to think network is the problem.

deth1k

so is traffic in question actually traversing VSL link? unlike a switch stack, traffic is not switched over backplane and traverses "traditional" port channel.

p.s i can understand your point re:CCIE however those four letters don't always mean anything.

gorebrush

deth1k wrote: »

Not every CCIE has experience in configuring / troubleshooting VSS, it's like asking a mechanic to fix F1 car. Why not approach the guy and ask what exactly is he doing?

That was the thought I had. I am close to sitting a lab and I wouldn't know how to confgure VSS. (v4 written I passed, btw)

gorebrush wrote: »

That was the thought I had. I am close to sitting a lab and I wouldn't know how to confgure VSS. (v4 written I passed, btw)

He was not asked to configure VSS, I did that months ago and its been working fine. he was asked to check out the network to see if it was the cause of poor performance on a storage issue. I would expect any one be they a CCENT or a CCIE that if they are looking at a network and they come across some thing they have no experienced before to check it out before they suggest making fundamental changes to the config/setup.

You are saying that if you came (paid) to troubleshoot an issue running across my core, and you sat down logged on and found that you were seeing both chassis presenting as a single logical switch and where told it was running as a VSS pair, you would not either take some time to read a whitepaper / documentation on the technology or at least ask me (the person who does know how to configure it) for the basics before claiming to have a solution.

I was asked to look at some MPLS the other day, my response was "I haven't worked with MPLS, let me take this away and have a play and I will get back to you". I don't mind someone not knowing, I don't like someone throwing ideas at a problem without knowing.

NOC-Ninja

I agree with the OP. Just want to add that VSS was not in the old "journey".

I suggest you ask him next time to explain to you what he is trying to do. I dont see anything wrong on asking.

NOC-Ninja wrote: »

I agree with the OP. Just want to add that VSS was not in the old "journey".
I suggest you ask him next time to explain to you what he is trying to do. I dont see anything wrong on asking.

99.9% of the issues in IT are not covered in any certification, this is why experience is important as is attitude to troubleshooting, indeed more important than any certifications someone might have. And we have discussed many time, we have conference call galore about the underlying issue that we are troubleshooting. And even after discussing my concerns the response was "well cant we try it anyway"

Something not being on a Certification course is no excuse for giving bad advice, especially when you are advising against fundamentally best practices. I have been lucky enough in my time to work with some truly gifted network engineers, and what I have seen without fail is that the letters CCIE have no relevance to if they are good or not. That is not to say I have not worked with some amazing CCIE's but I have also worked with some that are put to shame by guys with only a few years in the field.

Again in this case I am not there to question him and ask him not to break the network, but if I a company did not have a "decent" network engineer to question him then he might have gone ahead and carried this out causing untold issues to our network. And that is the issue, not knowing is find, putting a customers network at risk is unforgivable.

N2IT

Certs don't mean anything why is that even in the discussion. Either you can deliver or you can't. I agree ask the guy his strategy and see what he responds with.