Systems Vs. Networking

martaw

This thread is more geared toward shops large enough to have dedicated systems and dedicated networking staff.

I don't know if it is just me... but it seems like that unless a company is building out/upgrading etc. their network, you pretty much set up a network and it usually works. With systems, people install apps on servers. You have to deal with users. It just seems like being on the systems/server side of things gets more headaches than on the networking side of the house (see above disclaimer).

Has anyone else noticed that day-to-day working with servers lends itself to chewing up more of a sysadmin syseng's time vs day-to-day network admin/eng tasks?

jdballinger

You sir, have obviously never administered a good sized network.

If things worked the way they were supposed to all the time, there wouldn't be ANY network admins! Of course things break! You can always optimize a network, someone always needs something done. Just because an infrastructure is in place doesn't mean that there is nothing left to do with it!

cisco_trooper

That and the systems people always want to blame the network which causes a decent amount of tail chasing.

Iristheangel

Networking can encompass so much - security, wireless, routing, voice, etc. All of these things may have constant issues, upgrades, tweaking, etc. I can't tell you how many times a day we get tickets for phone issues or even someone wanting to add a network printer to a new port (which involves finding the physical port on that switch, enabling it, putting it on the right vlan, and adding a description/modifying documentation). Lets not forget all the security issues that come along, QoS, hardware failures, vpn, etc. Setting up the networks is the fun part... maintaining is the harder work

spicy ahi

cisco_trooper wrote: »

That and the systems people always want to blame the network which causes a decent amount of tail chasing.

Speaks the truth, he does. -Yoda

m3zilla

jdballinger wrote: »

You sir, have obviously never administered a good sized network. !

Exactly!

Small/Medium size network are usually pretty static in a sense that it doesn't change much. You set it up, create vlans, set up routing, you're good to go. In larger environment, there are many moving parts and new environment/connections constantly being brought up.

WiseWun

I work closely with the network team and any problem that arises, these guys get the blame/ requested first on a tech bridge.

Zartanasaurus

cisco_trooper wrote: »

That and the systems people always want to blame the network which causes a decent amount of tail chasing.

Yup. The network is a black box to the systems guys and it's ALWAYS the fault of the network or the firewall. This is especially true with vendor support for apps.

WiseWun

I work closely with the network team and any problem that arises, these guys get the blame/ requested first on a tech bridge.

phoeneous

As systems grow, so does the network. Just sayin.

ajs1976

Zartanasaurus wrote: »

Yup. The network is a black box to the systems guys and it's ALWAYS the fault of the network or the firewall. This is especially true with vendor support for apps.

then there are the times when after 1 1/2 years, network team finally fixes the 100 meg bottle necks and firewall packet inspection rules that were causing the problem in the first place.

shodown

Iristheangel wrote: »

Networking can encompass so much - security, wireless, routing, voice, etc. All of these things may have constant issues, upgrades, tweaking, etc. I can't tell you how many times a day we get tickets for phone issues or even someone wanting to add a network printer to a new port (which involves finding the physical port on that switch, enabling it, putting it on the right vlan, and adding a description/modifying documentation). Lets not forget all the security issues that come along, QoS, hardware failures, vpn, etc. Setting up the networks is the fun part... maintaining is the harder work

Yep. Only in the maintaining did I ever want to pull my hair out especially at the tier 3 level.

martaw

jdballinger wrote: »

You sir, have obviously never administered a good sized network.

Au contraire, mon frere... I worked for a datacenter service provider with multiple datacenters nationwide and that offered managed services (networking and systems) to multiple enterprise customers. The networking team was responsible for customer networks/troubleshooting AND internal company infrastructure. This included colo customers as well as well as our cloud/virtulization customers. Major LOL on above comment.

Take a new customer install of 100 VMs... Networking team provides firewall, switch, and possible load balancer setup. Systems team would have to provision all of their 100VMs... sometimes migrating all their data and even importing their VMs. Many times they would have to stay on weekends to make the deadline. Oh don't forget installing all their agents and all the QA. Networking install doesn't take nearly as long and is more held up by customer not providing all requirements more than anything.

Take patches and updates... Networking team would have a yearly code review of customer and internal infrastructure. Systems team would get random calls from customers with only hours notice to patch their bajillion servers THAT night.

Take monitoring calls from the NOC. Systems team would get calls on a huge array of systems alerts... from high CPU and memory utilization, to services not running, to whatever. Networking team rarely got a call in the middle of night. Occassionally there might be a customer VPN down (usually customer issue)... but usually if it HAD been working before and no changes have been made it stays working. Now granted the NOC should have been able to handle more... but it is what it is.

Many times customer would complain about their app performance. Sure networking would get blamed a lot... but customers LOVED to blame the underlying operating system. Seemed a lot easier to prove it wasn't the network to a customer. Setup a packet capture for source and destination, review the SYNs, ACKs, etc and clearly be able to show the customer the response time of their server is the issue many times. Systems team would get bogged down in all this research about this and that OS settings to finally try to prove to the customer that a problem is not with the OS.

The network architect there was very proud of all the scripting that he had in place to take a lot of manual work out of finding if there were any problems in the network. Other members had a bunch of scripts for customer installs. Enter a little data input, paste script on device and go.

OK, to irishangel... maybe VOIP is a different animal... but I really felt sorry for the systems guys.

m3zilla

So what is your role with the company? Sys admin or network?

I'm not sure how big/complex your network is, but where I work, we have a team of 20+ network engineers, and we stay plenty busy.

networkjutsu

Don't forget about packet analysis! Sometimes it takes a whole day to find a needle in a haystack. I worked for a company that had 4 - 6 full time packet guys and three network teams (Architects, Engineers & Technicians). Yup, they're not busy at all.

Apollo80

Just as systems need to be patched, upgraded, and replaced with newer hardware, so does the network infrastructure.

it_consultant

Oh boy, this argument again. Almost every network engineer that I know that doesn't work for an ISP wears more than one hat, i.e. many are also voice engineers. Inside corporate networks, no news is usually good news, if your network engineering team is always busy this is usually a sign of a needlessly complex network or a poorly set up network. The latter happens, there are a lot of CCNAs and CCNPs who are very bad network engineers.

Systems is very complex, I am not saying that networking is NOT complex, just that with systems there are more opportunities to screw something up. I can count on two hands how many times in my career a network change was solely to blame for a systems outage. Usually it was an innocent enough mistake (turning on IGMP snooping, mis-applied ACL, etc) which only required looking at the 'last change' to figure out where the problem lay.

Every system admin should be familiar with the concept of packet analysis to troubleshoot systems problems. If you are trying to connect to a something and you get a generic connection error, assuming you aren't jumping a firewall, it is normally never the switches in between the systems. Sharking the connection will let you know specifically where the error occurred which can help you configure the system correctly.

Every sysadmin should also be a mini-network engineer but the opposite is not always necessary for network engineers. What isn't helpful is the constant finger pointing. I get this with our DBAs and sysadmins. "Its not my problem its your problem"...guess what..it is all of our dang problem.

networkjutsu

I would disagree. There are companies out there that still operate in silos. Though, I have to say it is changing. The position that I was in was purely routing and switching. No other technology was involved. Sure, we had access to Infinistream Console, ACE Analyst and ACE Live, and Check Point Firewall (read only for network guys) but they weren't their main job. The Tier 1 (Network Technicians/Analysts) did have to learn basic troubleshooting for voice, wireless, VSAT, and many more though since they're the first line of defense (networking wise) but for the most part the core is still routing and switching since there are Tier 2 departments that will handle voice, wireless, and firewalls.

The network departments in that company were always busy because there were always new projects, existing projects that are now considered not-so-important anymore, projects that are really needed to improve the network infrastructure, equipment upgrades and IOS upgrades, and also supporting other IT teams. Though, I would agree that poorly designed and complex network would also cause network departments to be busy.

networker050184

In the end it comes down to everyone thinks the other one is no big deal!

martaw

m3zilla wrote: »

So what is your role with the company? Sys admin or network?

I'm not sure how big/complex your network is, but where I work, we have a team of 20+ network engineers, and we stay plenty busy.

I was part of networking. The networking and systems teams had around 15 or so people each but also divided up into sub-teams.

I will say this... I don't quite recall where I said that networking people are not busy. I did not say that I was not busy. Just to repeat... if you are involved in continual hardware/software upgrades and planning/implementing large new networks... then yeah networking people can be overloaded...

It's just my observation that you typically need a lot more server people than networking people simply because there are way more servers than networking gear to keep up.

rsutton

I am a Systems Admin, and I blame the network team every chance I get.

networkjutsu

Here's my experience

Company X

Network
Tier 1 (Network Technicians/Analysts) - 25+ people
Tier 2 (Network Engineers + Packet guys) - 15+ people
Tier 2 (Wireless Network Engineers) - 4 - 6 people
Tier 3 (Network Architects) - 4 - 6 people

Network Security (If you consider this as network team)
Around 5 - 8 people

Server Team
I'd say less than 15 people. I can't remember the exact number but way less than the network team.

Company Y

2 x Network guys

1 x System Admin

Company Z

Network team - 15+ people
System admin - 4 - 5 people

jamesp1983

Zartanasaurus wrote: »

Yup. The network is a black box to the systems guys and it's ALWAYS the fault of the network or the firewall. This is especially true with vendor support for apps.

Just as I was reading this I received a "We can't figure out what else it could be" alert from the server guys. Job security...

Zartanasaurus

jamesp1983 wrote: »

Just as I was reading this I received a "We can't figure out what else it could be" alert from the server guys. Job security...

A vendor made some software changes to one of our applications over the weekend and coincidentally, the application stopped working. The director came over and asked if there was anything on the network that was stopping it from working...

networkjutsu

Zartanasaurus wrote: »

A vendor made some software changes to one of our applications over the weekend and coincidentally, the application stopped working. The director came over and asked if there was anything on the network that was stopping it from working...

So typical. Where I used to work, we hear it every single day. That's why they have four to six full time packet guys who constantly just use OPNET or Infinistream Console. Next time, you should send them this picture.

atorven

From a small shop perspective, there is not much to do on the network side once it's up and running, not many outages either when compared to the system side of things.

phaneuf1

open a port for this, activate that lan drop, create a new vlan for that group of people, etc. Trust me, you'll always have jobs in network admin.

nethacker

Zartanasaurus wrote: »

A vendor made some software changes to one of our applications over the weekend and coincidentally, the application stopped working. The director came over and asked if there was anything on the network that was stopping it from working...

This is where i question the intelligence of some directors or IT managers. EVERYTHING is always the network or Firewall. so annoying when i have to troubleshoot for 1 hour and the issue is with the application server or application itself. that's the fun part though

networkjutsu

Where I used to work, network is expected to provide proof that it isn't a network issue. Hence, the 5-6 FTE dedicated to do just packet analysis.

FW guys don't even check the logs properly and expect network to answer every single network problem. That was the whole reason why we needed read-only access to the Check Point FWs to double check their investigation skills because there were several instances where it was after all a FW config issue.

nethacker

True that