Network Teams / Departments - You aren't wizards !

in Off-Topic
Possibly a rant, but in the last few years I have been through several companies which usually had their dedicated network teams. My roots are in MSPs where I was usually responsible for the end-to-end implementation of vSphere cluster. From ordering kit, configuring APCs, taking Cisco kit out of the box and configure the stacks and basics such as OSPF, VLANs, ACLs and whatnot ..
Time I needed to do that: A few days by myself - usually slowed down by travel between DCs.
Back to the point ... a few times now when I requested certain things, like adding a VLAN to an allow list, changing an accessport to trunk with native VLAN etc. - it usually takes WEEKS to implement. That isn't just one company I worked for - it's every single one with dedicated teams.
I needed a firewall port opened - took three weeks ... and the list goes on. I obviously understand that certain procedures have to be followed, like CAB etc., but really ?
One example. I was working in a finance company where I had to implement a new cluster. The switch was not connected to any production kit but was an isolated standalone 3750G which was pretty much just a **** managed switch. Only reason we used that is because we needed trunk ports, otherwise we would have even used a bloody desktop switch.
Now the network team insisted that I needed to raise tickets / CAB requests in order to add VLANs - that is totally fine .. but again - took days if not weeks to add a VLAN.
At some point I got peed off and read the switch details using CDP and didn't just ask for xyz, but even submitted the config ... still ... felt like walking against walls.
In another job the network team was not busy (we all knew ongoing projects of each teams) and yet a simply firewall rule took weeks. To a point where the business complained. The head of department did that immediately and was praised for his outstanding work ..
That goes on and on ...
One thing is very clear to a lot of the people in the business - some people make certain stuff harder than it should be to look good - sorry - but its true .... Especially seeing that most network guys browsed the internet all day.
When I had to add a stretched VLAN between multiple sites, I had to make sure I configure STP correctly, had to jump through countless routers and switches etc.
I know it doesn't take a month to do ... so don't hang out on youtube all day and tell me it is hard work and takes a month.
Latest fun : I had to request several ports (GbE and 10GbE). Bear in mind - network team does not deal with the fabric, all I need is an allocation of the ports, have them configured as trunk and add a single firewall rule.
Bear in mind - cabling is done by my team, fabric is done by my team - all we require is indeed a handful ports to be configured plus a single firewall rule.
Estimate I received from the team ? 20 MANDAYS .... that's EFFORT MANDAYS ....
Really ? It takes 4 weeks of constant typing in order to change trunks ....
Seriously, I wish some people would stop selling networks as black magic, otherwise I start estimating a VM clone at 30 ManDays ...
Time I needed to do that: A few days by myself - usually slowed down by travel between DCs.
Back to the point ... a few times now when I requested certain things, like adding a VLAN to an allow list, changing an accessport to trunk with native VLAN etc. - it usually takes WEEKS to implement. That isn't just one company I worked for - it's every single one with dedicated teams.
I needed a firewall port opened - took three weeks ... and the list goes on. I obviously understand that certain procedures have to be followed, like CAB etc., but really ?
One example. I was working in a finance company where I had to implement a new cluster. The switch was not connected to any production kit but was an isolated standalone 3750G which was pretty much just a **** managed switch. Only reason we used that is because we needed trunk ports, otherwise we would have even used a bloody desktop switch.
Now the network team insisted that I needed to raise tickets / CAB requests in order to add VLANs - that is totally fine .. but again - took days if not weeks to add a VLAN.
At some point I got peed off and read the switch details using CDP and didn't just ask for xyz, but even submitted the config ... still ... felt like walking against walls.
In another job the network team was not busy (we all knew ongoing projects of each teams) and yet a simply firewall rule took weeks. To a point where the business complained. The head of department did that immediately and was praised for his outstanding work ..
That goes on and on ...
One thing is very clear to a lot of the people in the business - some people make certain stuff harder than it should be to look good - sorry - but its true .... Especially seeing that most network guys browsed the internet all day.
When I had to add a stretched VLAN between multiple sites, I had to make sure I configure STP correctly, had to jump through countless routers and switches etc.
I know it doesn't take a month to do ... so don't hang out on youtube all day and tell me it is hard work and takes a month.
Latest fun : I had to request several ports (GbE and 10GbE). Bear in mind - network team does not deal with the fabric, all I need is an allocation of the ports, have them configured as trunk and add a single firewall rule.
Bear in mind - cabling is done by my team, fabric is done by my team - all we require is indeed a handful ports to be configured plus a single firewall rule.
Estimate I received from the team ? 20 MANDAYS .... that's EFFORT MANDAYS ....
Really ? It takes 4 weeks of constant typing in order to change trunks ....
Seriously, I wish some people would stop selling networks as black magic, otherwise I start estimating a VM clone at 30 ManDays ...
My own knowledge base made public: http://open902.com 

Comments
While it's possible that the teams you're working with are just slow, lazy, incompetent, etc., sometimes network teams are really busy juggling multiple projects, troubleshooting an edge case that's taking much longer than expected, or reviewing the change request with sufficient depth to ensure it doesn't cause unintended side effects.
(It's exactly the same in my company)
I get all that - but like I say - we are talking EFFORT MANDAYS - not the time it takes going through hoops / procedures etc.
"Hey guys, the gig is up, we can open the curtain now." (This is a reference to the Wizard of Oz being "behind the curtain")
On topic, this seems to be an issue of the change management process, not necessarily an issue with the network team.
If it's anything like where I worked in the past, changes for certain things had to occur at certain dates and certain times. Whether it took 30 seconds to implement or 30 minutes, we had to follow the change control process.
Fortunately for us, the change control process was integrated into our ticketing system, and changes requested that morning would often be signed off on and implemented that evening (if maintenance windows were available for the concerned devices at that time). Other times, they would occur in the next day or two. For some of our more important sites, we may have only weekly maintenance windows.
I dare say that you have more of a beef with your change control process and the current service level agreements than your network team.
LinkedIn: http://www.linkedin.com/in/lewislampkin (Please connect: Just say you're from TechExams.Net!)
Had an issue with performance for remote users. They continually blamed the Citrix Access Gateways even though it pointed to a firewall issue. A year later, they looked into the firewalls causing the problem and found it.
2017 Goals: 1 of 5 courses complete, 0 of 2 exams complete
IPSec VPN Design 44%
Mastering VMWare vSphere 5 42.8%
I said it before - we are talking about EFFORT MANDAYS. Just to explain what that means.
You have a project. Say, implementing X - you estimate 10 mandays for this - that means YOU need to WORK on this for 10 days - 8hrs each - 80hrs.
This EFFORT does not include any delays caused by processes, procedures, meetings, management delays and the sorts - all these factors have their own required time associated.
When I say it takes them 20 Mandays to implement these configurations, then it simply means they need 160hrs to perform the change on the switch as by this point all processes, procedures, have passed. In fact in this case there are an additional 5 Days (EFFORT MANDAYS) scheduled for the DESIGN.
So he estimates it takes at least 40hrs to design a port change.
And it annoys me so much because it is the same old in every company I worked for.
If someone can say here by heart that it can easily take 40hrs to design a config change, 160hrs to perform the change and additional days for "support" - then please PLEASE explain me how this is possible and I shut up ...
FW changes where 7 days, of which 5 where taken up by a change management process that was out side of the teams control, and for core FW a 14 day process. This was because mistakes did not cost £100's but £1,000,000 an hour!
Funny how us network guys say the same things about the storage and server guys. takes 6 months to implement a back up solution!! or 2 months to issue a enterprise Cert. honestly I can do this stuff in my sleep, but I get sob stories of how complicated it is and that it needs to be planned. As for VMware I got so fed up waiting for the VMware guy to do it that I gave up and just informed management I would do it my self.
This is not a Network or server guys issue, its just poorly managed and unprofessional staff. It happens across the board and has nothing to do with the lenth of time it takes to carry things out correctly, it all about setting peoples expectations and keeping them updated when things are happening and if there are delays why they are happening.
And honestly I think some one is getting SLA and mandays confused with in the network team!!
I have had CCNP's look at me blankly when I say you 'tag' VLANs and 'trunk' port channels - because thats what you do. Calling it 'trunking' mode is a mistake Cisco made years ago that no one ever rectified. Moreover, if you ask what a port in trunk mode actually means/does, I often get a blank stare. If you don't understand how tagged frames are processed by the switch, when we use "dual-mode" to work with a VOIP phone, or use the native VLAN plus a tagged VLAN - of course you are going to get a blank stare.
It goes both ways and usually comes down to politics or work prioritization. You aren't getting anything done in less than a week or two (a lot of times that is pushing it) in almost every place I've worked unless you have either director level escalation or you are in good with someone willing to do it on the spot.
Casual reading: CCNP, Windows Sysinternals Administrator's Reference, Network Warrior
This is just an indication you're swallowing a different vendor's Kool-Aid. It was not only Cisco who used the terms "trunk" and "access" links prior to 2004, but also the IEEE. See 802.1Q-1998, for example. Following the standard is hardly a mistake, and starting in the market later and thus using more modern terms, is a paper-thin competitive advantage.
Obviously, knowing the different terms and perspectives--Cisco (trunk, native, access) vs Others (PVID, tagged, untagged, mixed)--is quite handy, especially if you're operating in a multi-vendor environment. If you're operating in a purely or primarily Cisco environment, I would use Cisco terms when you can, to limit the potential for confusion.
You shouldn't get a blank stare from a CCNP. Most CCNPs would have an easy time with this since the topic's covered extensively. If you do, it's an indication of a paper-CCNP, probably braindumps, and poor hiring practices.
This you may have to explain to them, since R&S only very lightly touches VoIP. That's a separate track.
If I hear a R&S guy ever tell me that they can't do a dual-mode to make a VOIP phone work they will lose all gravitas with me. I understand Cisco doesn't cover it (there is a lot to networking and not everything is covered in certs) but this is a fairly basic task. The problem isn't that it isn't covered in the certs it is the basic lack of understanding of what trunked mode is in the first place. I used that as a specific example because OP mentioned pruning or allowing VLANs on a trunk as a task that would take a long time at his job. Indeed I have seen blank stares when I have talked to CCNPs when I said "we can tag the link to Vsphere or whatever" when they were expecting to hear "trunk". Of course I am not going to say trunk because (esp if you are connecting to LACP NICs) trunks are a totally different thing. Regardless of the nomenclature, a good network engineer can simply translate that in their head. It is a sign they have been on heterogenous networks and wont get thrown by simple things.
It comes down to this, if I tell a network engineer that I need his device to be LACP active, he should know whatever commands are necessary to make that happen on their platform. If you are thrown by the fact I didn't say 'mode desireable' or whatever the command is (I think that is actually telling the switch to do PaGP), or 'etherchannel' then you are totally useless to me.
I don't see this in the storage world, for example. Someone who works on an EMC SAN with Brocade FC switches usually can easily transition to Hitachi with Cisco or QLOGIC switches.
It isn't that all networking guys are bad, far from it, but I certainly have seen my fair share who when tossed into an actual network that hasn't already been optimized or standardized; they fall flat on their faces. Remember that I come primarily from the consulting world where I worked with a ton of network engineers, my sample size isn't HUGE but it is fairly diverse.
What I do relies though is the good windows engineers understand windows and the surrounding tec to a level I can only dream of archiving. So no networking is not a dark art, but to be honest in general network engineers have a better knowledge of other technologies through nesisity than non network/security engineers have of networking.
Fine slate Lazy useless staff, but like all areas there are good and bad engineers and I have worked in plenty of companies with rubbish server admin, and crap software development teams. Let's show a bit of respect for people who spend there life working to archive understanding across all fields in IT and stop silly ill thought through finger pointing.
I'm in a small team of 5 - 1 Network Architect, 1 Manager, 1 Data Engineer (Me), 2 Telecom people...
If we need a change done, our director and business heads trust us 100% but it's usually just a week wait for a change request.
I couldn't possibly give someone a 4 week wait time for a VLAN change... What the heck would I do with myself?
Browse reddit and techexams all day, maybe?
You're busy with all the other requests that were put in four weeks ago and have finally come for their time to be implemented.
Blog: www.network-node.com
CISCO
"A flute without holes, is not a flute. A donut without a hole, is a Danish" - Ty Webb
Reading:NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures
Again, it seems that people don't read the whole post or responses. Like I said a few times now - I don't argue with SLAs, change control or documentations, I am talking about the EFFORT in performing a port change. Let me try again to explain my issue here in more detail.
In our company, each change has to go through change control. In bigger projects each team has to submit estimates for time and cost and basically a full plan according to the SDLC.
The EFFORT described does NOT include time needed for change control as the teams usually don't have an influence in that.
Example. Network Team is being asked to do exactly that, a port change within a project. So they have to submit times for
a. Design (HL / LL)
b. Risk Analysis and Mediation
c. Implementation
d. Testing / QA
Got it ?
Now in this instance, and that is just an example of many, we received the following estimates
1. HL Design (which cable plugs in where) - 5 Days
2. LL Design (which 4 VLANs are configured / allowed on the trunk port) - 5 Days
3. Risk Analysis (which was a one liner) - 2 Days
4. Implementation (Configure 6 ports to be trunk ports, create allow list of 4 VLANs) - 15 Days
5. Testing (which was again a one liner as final testing was done by system team) - 5 Days
So WITHOUT any change control / management sign off / review - 6 ports being enabled and configured as Trunk with 4 VLANs : 32 Days. These are Mandays with 8.5hrs each .. So we are talking about 256 FULL HOURS to do this ...
If you still say this is not lazy and reasonable then I give up ...
"Of course Aaron, but it will be about 8-12 weeks"
WFT!!! 8-12 weeks?
Oh there implementing a new backup solution they tell me, and can't create any new guests until it has been completed and tested!
If they where implementing a VMware upgrade or host upgrade then OK you need stability, but a backup solution means they cant create guests what kind of rubbish is that..
No one is saying that 256 hours to do a port change is absurd, although in some companies if you add all the change managers and engineers time together then 10+ hours for a port chance is quite reasonable (I worked some where that port changes had to be approved by 7 separate departments as company policy). but yer 256 is just a tad over the top for 99% of places.
The fact is though this has nothing to do with networking engineers, just bad management at your company. Happens across the board in IT departments. I could list of the issue I have had with the server guys, or the software developers (2 years to correct the spelling of the company name in a public facing web site).
Where I work the network is usually the fall-guy, the amount of dross that gets sent our way because desktops or the server guys dont do their troubleshooting properly is amazing. The wireless network has blackspots, oh wait, the user hadnt enabled wifi, why havent Desktops troubleshooted that before sending it to us? Very often the first time the network team is even aware of an IT project is when the first servicedesk ticket lands at our feet saying something isnt working.
Oh, and if we are on youtube its because we need to check that the gateway is still up!
I do not get it but I think I was almost there...please use more bold lettering and underlined words
CISCO
"A flute without holes, is not a flute. A donut without a hole, is a Danish" - Ty Webb
Reading:NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures
I think we all agree that it doesn't take 4 weeks of typing to change trunks... As others have said, they probably aren't explaining it right or just overestimating to make their jobs looks important.
You may learn something!