Network Teams / Departments - You aren't wizards !

jibbajabba

Possibly a rant, but in the last few years I have been through several companies which usually had their dedicated network teams. My roots are in MSPs where I was usually responsible for the end-to-end implementation of vSphere cluster. From ordering kit, configuring APCs, taking Cisco kit out of the box and configure the stacks and basics such as OSPF, VLANs, ACLs and whatnot ..

Time I needed to do that: A few days by myself - usually slowed down by travel between DCs.

Back to the point ... a few times now when I requested certain things, like adding a VLAN to an allow list, changing an accessport to trunk with native VLAN etc. - it usually takes WEEKS to implement. That isn't just one company I worked for - it's every single one with dedicated teams.

I needed a firewall port opened - took three weeks ... and the list goes on. I obviously understand that certain procedures have to be followed, like CAB etc., but really ?

One example. I was working in a finance company where I had to implement a new cluster. The switch was not connected to any production kit but was an isolated standalone 3750G which was pretty much just a **** managed switch. Only reason we used that is because we needed trunk ports, otherwise we would have even used a bloody desktop switch.

Now the network team insisted that I needed to raise tickets / CAB requests in order to add VLANs - that is totally fine .. but again - took days if not weeks to add a VLAN.

At some point I got peed off and read the switch details using CDP and didn't just ask for xyz, but even submitted the config ... still ... felt like walking against walls.

In another job the network team was not busy (we all knew ongoing projects of each teams) and yet a simply firewall rule took weeks. To a point where the business complained. The head of department did that immediately and was praised for his outstanding work ..

That goes on and on ...

One thing is very clear to a lot of the people in the business - some people make certain stuff harder than it should be to look good - sorry - but its true .... Especially seeing that most network guys browsed the internet all day.

When I had to add a stretched VLAN between multiple sites, I had to make sure I configure STP correctly, had to jump through countless routers and switches etc.

I know it doesn't take a month to do ... so don't hang out on youtube all day and tell me it is hard work and takes a month.

Latest fun : I had to request several ports (GbE and 10GbE). Bear in mind - network team does not deal with the fabric, all I need is an allocation of the ports, have them configured as trunk and add a single firewall rule.

Bear in mind - cabling is done by my team, fabric is done by my team - all we require is indeed a handful ports to be configured plus a single firewall rule.

Estimate I received from the team ? 20 MANDAYS .... that's EFFORT MANDAYS ....

Really ? It takes 4 weeks of constant typing in order to change trunks ....

Seriously, I wish some people would stop selling networks as black magic, otherwise I start estimating a VM clone at 30 ManDays ...

docrice

This really depends on the complexity of the network and change-control procedures for accountability and audit purposes. Simply adding a VLAN or doing firewall changes may have unpredictable results if not done right. I speak as someone on a network team involved in these sorts of things. I've also broken networks and caused downstream damage to hosts themselves because of a single mistake.

While it's possible that the teams you're working with are just slow, lazy, incompetent, etc., sometimes network teams are really busy juggling multiple projects, troubleshooting an edge case that's taking much longer than expected, or reviewing the change request with sufficient depth to ensure it doesn't cause unintended side effects.

gorebrush

It'll be the bureaucracy that is your problem here!

(It's exactly the same in my company)

jibbajabba

docrice wrote: »

This really depends on the complexity of the network and change-control procedures for accountability and audit purposes. Simply adding a VLAN or doing firewall changes may have unpredictable results if not done right. I speak as someone on a network team involved in these sorts of things. I've also broken networks and caused downstream damage to hosts themselves because of a single mistake.

While it's possible that the teams you're working with are just slow, lazy, incompetent, etc., sometimes network teams are really busy juggling multiple projects, troubleshooting an edge case that's taking much longer than expected, or reviewing the change request with sufficient depth to ensure it doesn't cause unintended side effects.

I get all that - but like I say - we are talking EFFORT MANDAYS - not the time it takes going through hoops / procedures etc.

docrice

Perhaps it's just "padding." I've heard of some places where a simple change has an SLA of extraordinary length because ... well, someone agreed to it without understanding what's really involved. As mentioned above, probably bureaucratic bloat.

instant000

Wow, not a wizard?

"Hey guys, the gig is up, we can open the curtain now." (This is a reference to the Wizard of Oz being "behind the curtain")

On topic, this seems to be an issue of the change management process, not necessarily an issue with the network team.

If it's anything like where I worked in the past, changes for certain things had to occur at certain dates and certain times. Whether it took 30 seconds to implement or 30 minutes, we had to follow the change control process.

Fortunately for us, the change control process was integrated into our ticketing system, and changes requested that morning would often be signed off on and implemented that evening (if maintenance windows were available for the concerned devices at that time). Other times, they would occur in the next day or two. For some of our more important sites, we may have only weekly maintenance windows.

I dare say that you have more of a beef with your change control process and the current service level agreements than your network team.

ajs1976

At my last job, common response from network team was "ping is good" and then they would disappear.

Had an issue with performance for remote users. They continually blamed the Citrix Access Gateways even though it pointed to a firewall issue. A year later, they looked into the firewalls causing the problem and found it.

Zartanasaurus

Guess it's time to stop putting on my robe and wizard hat.

jibbajabba

Like I said now a few times - change processes etc. isn't even included and I don't complain about that - CAB happens weekly so in theory there is a week delay in requests - again - if CAB delays it - fine - don't have an issue with that ..

I said it before - we are talking about EFFORT MANDAYS. Just to explain what that means.

You have a project. Say, implementing X - you estimate 10 mandays for this - that means YOU need to WORK on this for 10 days - 8hrs each - 80hrs.

This EFFORT does not include any delays caused by processes, procedures, meetings, management delays and the sorts - all these factors have their own required time associated.

When I say it takes them 20 Mandays to implement these configurations, then it simply means they need 160hrs to perform the change on the switch as by this point all processes, procedures, have passed. In fact in this case there are an additional 5 Days (EFFORT MANDAYS) scheduled for the DESIGN.

So he estimates it takes at least 40hrs to design a port change.

And it annoys me so much because it is the same old in every company I worked for.

If someone can say here by heart that it can easily take 40hrs to design a config change, 160hrs to perform the change and additional days for "support" - then please PLEASE explain me how this is possible and I shut up ...

DevilWAH

Dont know what company you work in, but port assignment when I was working network change management was 24hours and that included a globable change management process.

FW changes where 7 days, of which 5 where taken up by a change management process that was out side of the teams control, and for core FW a 14 day process. This was because mistakes did not cost £100's but £1,000,000 an hour!

Funny how us network guys say the same things about the storage and server guys. takes 6 months to implement a back up solution!! or 2 months to issue a enterprise Cert. honestly I can do this stuff in my sleep, but I get sob stories of how complicated it is and that it needs to be planned. As for VMware I got so fed up waiting for the VMware guy to do it that I gave up and just informed management I would do it my self.

This is not a Network or server guys issue, its just poorly managed and unprofessional staff. It happens across the board and has nothing to do with the lenth of time it takes to carry things out correctly, it all about setting peoples expectations and keeping them updated when things are happening and if there are delays why they are happening.

And honestly I think some one is getting SLA and mandays confused with in the network team!!

it_consultant

In my experience the delays in network changes are normally due to staff incompetence. For a lot of network guys, the network IS kind of a black magic dark room of mystery. I think it is primarily because a lot of the certifications focus too much on vendor add ons (I am looking at you Cisco and Brocade) and not enough on how the underlying stuff actually bolts together.

I have had CCNP's look at me blankly when I say you 'tag' VLANs and 'trunk' port channels - because thats what you do. Calling it 'trunking' mode is a mistake Cisco made years ago that no one ever rectified. Moreover, if you ask what a port in trunk mode actually means/does, I often get a blank stare. If you don't understand how tagged frames are processed by the switch, when we use "dual-mode" to work with a VOIP phone, or use the native VLAN plus a tagged VLAN - of course you are going to get a blank stare.

networker050184

Almost as bad as those systems guys that take months to make a couple clicks in a GUI! I mean they could do that in two minutes right?

It goes both ways and usually comes down to politics or work prioritization. You aren't getting anything done in less than a week or two (a lot of times that is pushing it) in almost every place I've worked unless you have either director level escalation or you are in good with someone willing to do it on the spot.

NetworkVeteran

networker050184 wrote: »

It goes both ways and usually comes down to politics or work prioritization. You aren't getting anything done in less than a week or two (a lot of times that is pushing it) in almost every place I've worked unless you have either director level escalation or you are in good with someone willing to do it on the spot.

+1. I usually need to ping a VP if I want those wily system guys to do anything pronto. I suppose this is a good place to share my story of going to the system team and having three separate people tell me a Unix folder couldn't be made world writable without being world readable. I had to escalate it to the director level before finally getting someone to run the simple chmod command.

elTorito

Makes me glad I work a small-ish IT environment where everything - and I mean everything - infrastructure-related is our team's responsibility. That includes server, virtualization, mail, network, SAN, storage, backup and firewall.

NetworkVeteran

it_consultant wrote: »

Calling it 'trunking' mode is a mistake Cisco made years ago that no one ever rectified.

This is just an indication you're swallowing a different vendor's Kool-Aid. It was not only Cisco who used the terms "trunk" and "access" links prior to 2004, but also the IEEE. See 802.1Q-1998, for example. Following the standard is hardly a mistake, and starting in the market later and thus using more modern terms, is a paper-thin competitive advantage.

Obviously, knowing the different terms and perspectives--Cisco (trunk, native, access) vs Others (PVID, tagged, untagged, mixed)--is quite handy, especially if you're operating in a multi-vendor environment. If you're operating in a purely or primarily Cisco environment, I would use Cisco terms when you can, to limit the potential for confusion.

Moreover, if you ask what a port in trunk mode actually means/does, I often get a blank stare.

You shouldn't get a blank stare from a CCNP. Most CCNPs would have an easy time with this since the topic's covered extensively. If you do, it's an indication of a paper-CCNP, probably braindumps, and poor hiring practices.

when we use "dual-mode" to work with a VOIP phone

This you may have to explain to them, since R&S only very lightly touches VoIP. That's a separate track.

it_consultant

NetworkVeteran wrote: »

This is just an indication you're swallowing a different vendor's Kool-Aid. It was not only Cisco who used the terms "trunk" and "access" links prior to 2004, but also the IEEE. See 802.1Q-1998, for example. Following the standard is hardly a mistake, and starting in the market later and thus using more modern terms, is a paper-thin competitive advantage.

Obviously, knowing the different terms and perspectives--Cisco (trunk, native, access) vs Others (PVID, tagged, untagged, mixed)--is quite handy, especially if you're operating in a multi-vendor environment. If you're operating in a purely or primarily Cisco environment, I would use Cisco terms when you can, to limit the potential for confusion.


You shouldn't get a blank stare from a CCNP. Most CCNPs would have an easy time with this since the topic's covered extensively. If you do, it's an indication of a paper-CCNP, probably braindumps, and poor hiring practices.


This you may have to explain to them, since R&S only very lightly touches VoIP. That's a separate track.

If I hear a R&S guy ever tell me that they can't do a dual-mode to make a VOIP phone work they will lose all gravitas with me. I understand Cisco doesn't cover it (there is a lot to networking and not everything is covered in certs) but this is a fairly basic task. The problem isn't that it isn't covered in the certs it is the basic lack of understanding of what trunked mode is in the first place. I used that as a specific example because OP mentioned pruning or allowing VLANs on a trunk as a task that would take a long time at his job. Indeed I have seen blank stares when I have talked to CCNPs when I said "we can tag the link to Vsphere or whatever" when they were expecting to hear "trunk". Of course I am not going to say trunk because (esp if you are connecting to LACP NICs) trunks are a totally different thing. Regardless of the nomenclature, a good network engineer can simply translate that in their head. It is a sign they have been on heterogenous networks and wont get thrown by simple things.

It comes down to this, if I tell a network engineer that I need his device to be LACP active, he should know whatever commands are necessary to make that happen on their platform. If you are thrown by the fact I didn't say 'mode desireable' or whatever the command is (I think that is actually telling the switch to do PaGP), or 'etherchannel' then you are totally useless to me.

I don't see this in the storage world, for example. Someone who works on an EMC SAN with Brocade FC switches usually can easily transition to Hitachi with Cisco or QLOGIC switches.

It isn't that all networking guys are bad, far from it, but I certainly have seen my fair share who when tossed into an actual network that hasn't already been optimized or standardized; they fall flat on their faces. Remember that I come primarily from the consulting world where I worked with a ton of network engineers, my sample size isn't HUGE but it is fairly diverse.

DevilWAH

Lets not start the server / network war. Yes I am a network engineer, but unlike mose server engineers not only can I configure complex networks, (including VoIP). I not only can I log on to both windows and Linux boxes with out help, I can also set up a full domain, exchange, apache, VMware, netapps, know a number of scripting languages and have a fair grasp of programming. Whilst most windows engineers I know would not know where to start with cisco IOS or Linux, if they can even log on with out help.

What I do relies though is the good windows engineers understand windows and the surrounding tec to a level I can only dream of archiving. So no networking is not a dark art, but to be honest in general network engineers have a better knowledge of other technologies through nesisity than non network/security engineers have of networking.

Fine slate Lazy useless staff, but like all areas there are good and bad engineers and I have worked in plenty of companies with rubbish server admin, and crap software development teams. Let's show a bit of respect for people who spend there life working to archive understanding across all fields in IT and stop silly ill thought through finger pointing.

phoeneous

Network ninja > wizard

darkerz

That's interesting,

I'm in a small team of 5 - 1 Network Architect, 1 Manager, 1 Data Engineer (Me), 2 Telecom people...

If we need a change done, our director and business heads trust us 100% but it's usually just a week wait for a change request.

I couldn't possibly give someone a 4 week wait time for a VLAN change... What the heck would I do with myself?

Browse reddit and techexams all day, maybe?

it_consultant

This is similar to my organization. Large IT departments can hide bad people.

networker050184

darkerz wrote: »

I couldn't possibly give someone a 4 week wait time for a VLAN change... What the heck would I do with myself?

You're busy with all the other requests that were put in four weeks ago and have finally come for their time to be implemented.

Iristheangel

It all depends on priority. It may seem like an easy task to add a VLAN or enable a port but I find that other teams often don't have any idea what you have on your plate. I might have a dozen or so such requests on my desk but working on a priority project for the CIO with a hard deadline of a week and furiously configuring routers/switches to get mailed out to a new site, finishing my documentation for another project that is due on my bosses desk at the end of the day, or trying to finish AP/IDF/MDF placements on 20 floorplans for our wiring vendor so they can immediately start wiring. When it comes to the small network adjustments like that, if it's not affecting a group of users or impacting the business as a whole, it's usually pushed down on my priorities and I'll get to it when I'm able to. I'm not going to stop a huge project with a near deadline to go through my 5 requests for VLANs or to enabled a port in some random conference room unless my boss instructs me too. I'm sure he would rather have a whole site have their network up by their move-in date and have that one VLAN pushed off for a few days

VAHokie56

Some times network changes are huge and involve there own design documents that have to presented to review boards that have every faction of the enterprise environment represented. After that said design equates to change orders that have to be approved and checked. If your network crew runs a tight ship they may even do a sort of peer review process to make sure you config scripts are up to snuff. We do all this so we can have tight SLA's 99.9% up time...SO next time maybe think about all that before shacking a fist at your network guy

Mrock4

I was the lead engineer in a place like you described. Prior to being the lead, people hated us, and avoided calling us. Once I became the lead I instituted a legit "buck stops here" mentality among all engineers (it took a little bit) - no more blame games, no tying people up in red tape. In only a couple of months we went from nobodies to having fantastic relationships with other departments, began getting recognized for excellence, and everyone enjoyed their jobs more. Not to mention, we eliminated as much red tape as we could. I have since left that job, but I'm proud to say that to this day, the engineers still seek problems out when dealing with other departments (many engineers search for proof there isn't a problem, they search for proof there IS..in order to facilitate a resolution).

CodeBlox

We have no change management where I work. I'm on our network team and when it comes time to make changes on the network, I carefully plan and document it. I like things this way, so much quicker getting stuff done. Then again, we are not a large shop (We really should be though). 3 admins and 2 engineers on our network team.

jibbajabba

VAHokie56 wrote: »

Some times network changes are huge and involve there own design documents that have to presented to review boards that have every faction of the enterprise environment represented. After that said design equates to change orders that have to be approved and checked. If your network crew runs a tight ship they may even do a sort of peer review process to make sure you config scripts are up to snuff. We do all this so we can have tight SLA's 99.9% up time...SO next time maybe think about all that before shacking a fist at your network guy

Again, it seems that people don't read the whole post or responses. Like I said a few times now - I don't argue with SLAs, change control or documentations, I am talking about the EFFORT in performing a port change. Let me try again to explain my issue here in more detail.

In our company, each change has to go through change control. In bigger projects each team has to submit estimates for time and cost and basically a full plan according to the SDLC.

The EFFORT described does NOT include time needed for change control as the teams usually don't have an influence in that.

Example. Network Team is being asked to do exactly that, a port change within a project. So they have to submit times for

a. Design (HL / LL)
b. Risk Analysis and Mediation
c. Implementation
d. Testing / QA

Got it ?

Now in this instance, and that is just an example of many, we received the following estimates

1. HL Design (which cable plugs in where) - 5 Days
2. LL Design (which 4 VLANs are configured / allowed on the trunk port) - 5 Days
3. Risk Analysis (which was a one liner) - 2 Days
4. Implementation (Configure 6 ports to be trunk ports, create allow list of 4 VLANs) - 15 Days
5. Testing (which was again a one liner as final testing was done by system team) - 5 Days

So WITHOUT any change control / management sign off / review - 6 ports being enabled and configured as Trunk with 4 VLANs : 32 Days. These are Mandays with 8.5hrs each .. So we are talking about 256 FULL HOURS to do this ...

If you still say this is not lazy and reasonable then I give up ...

DevilWAH

No different to me asking the VMware team to create a new guest with out Redhat template image on it for me to install a network management tool.

"Of course Aaron, but it will be about 8-12 weeks"

WFT!!! 8-12 weeks?

Oh there implementing a new backup solution they tell me, and can't create any new guests until it has been completed and tested!

If they where implementing a VMware upgrade or host upgrade then OK you need stability, but a backup solution means they cant create guests what kind of rubbish is that..

No one is saying that 256 hours to do a port change is absurd, although in some companies if you add all the change managers and engineers time together then 10+ hours for a port chance is quite reasonable (I worked some where that port changes had to be approved by 7 separate departments as company policy). but yer 256 is just a tad over the top for 99% of places.

The fact is though this has nothing to do with networking engineers, just bad management at your company. Happens across the board in IT departments. I could list of the issue I have had with the server guys, or the software developers (2 years to correct the spelling of the company name in a public facing web site).

cisco_dog

If you have experienced this at every company you've for maybe you are the common problem :wink

Where I work the network is usually the fall-guy, the amount of dross that gets sent our way because desktops or the server guys dont do their troubleshooting properly is amazing. The wireless network has blackspots, oh wait, the user hadnt enabled wifi, why havent Desktops troubleshooted that before sending it to us? Very often the first time the network team is even aware of an IT project is when the first servicedesk ticket lands at our feet saying something isnt working.

Oh, and if we are on youtube its because we need to check that the gateway is still up!

VAHokie56

jibbajabba wrote: »

Again, it seems that people don't read the whole post or responses. Like I said a few times now - I don't argue with SLAs, change control or documentations, I am talking about the EFFORT in performing a port change. Let me try again to explain my issue here in more detail.

In our company, each change has to go through change control. In bigger projects each team has to submit estimates for time and cost and basically a full plan according to the SDLC.

The EFFORT described does NOT include time needed for change control as the teams usually don't have an influence in that.

Example. Network Team is being asked to do exactly that, a port change within a project. So they have to submit times for

a. Design (HL / LL)
b. Risk Analysis and Mediation
c. Implementation
d. Testing / QA

Got it ?

Now in this instance, and that is just an example of many, we received the following estimates

1. HL Design (which cable plugs in where) - 5 Days
2. LL Design (which 4 VLANs are configured / allowed on the trunk port) - 5 Days
3. Risk Analysis (which was a one liner) - 2 Days
4. Implementation (Configure 6 ports to be trunk ports, create allow list of 4 VLANs) - 15 Days
5. Testing (which was again a one liner as final testing was done by system team) - 5 Days

So WITHOUT any change control / management sign off / review - 6 ports being enabled and configured as Trunk with 4 VLANs : 32 Days. These are Mandays with 8.5hrs each .. So we are talking about 256 FULL HOURS to do this ...

If you still say this is not lazy and reasonable then I give up ...

I do not get it but I think I was almost there...please use more bold lettering and underlined words

jibbajabba

ok, will do.

Mishra

jibbajabba wrote: »

Really ? It takes 4 weeks of constant typing in order to change trunks ....

I think we all agree that it doesn't take 4 weeks of typing to change trunks... As others have said, they probably aren't explaining it right or just overestimating to make their jobs looks important.