Daily duties of a CCNA

lboppi

I recently took a position of an IT director. In the past I worked as Helpdesk technician, desktop technician, and Microsoft System Administrator and IT Project manager. Now finally I took the IT Director position few weeks ago.

As you can see my background is on Windows platform and I have very little knowledge on Cisco side.

Under my supervision we have 2 Cisco Administrators. In the past I have notice that these 2 administrator hardly do anything. They will chat online, play in face book, stop office to office and chat and do nothing. When I was their coworker I jokingly said many time “ what do you guys do all day” , and their reply was “ everything working fine, we have nothing to do, if you see a problem , let us know” .

Now that I am their boss, I want to make sure they are doing what they supposed to. From my Windows system administrator days I know even if every thing is working there are lot a things to do such as – updates, monitor servers , or start finding things to improve.

Here is our network environment. We have 200 remote locations which all connect to our central office via VPN. Each location has cisco ASA, PIX and router and in our central location we have VPN concentrator.

My question is to all CCNA – in your work what do you do every day? If everything working as it should does that mean there is nothing to do on Cisco network? How often cisco release firmware updates? How often do we need to take a look at entire network infrastructure and if we need to redesign the network.

petedude

lboppi wrote: »

My question is to all CCNA – in your work what do you do every day? If everything working as it should does that mean there is nothing to do on Cisco network? How often cisco release firmware updates? How often do we need to take a look at entire network infrastructure and if we need to redesign the network.

Congrats on the promotion.

I would think these gentlemen should be performing periodic (daily?) health checks on the Cisco devices, not to mention periodically checking logs to ensure that there are no security breaches. They should also be monitoring traffic-- are there SLAs governing expected bandwidth to your users? If so, these guys need to be watching for movie downloaders, etc.

lboppi

Thanks for the reply. Well we have Solar Wind NPM and Palo Alto Firewall to monitor traffic, syslogs, traps etc. They are monitoring these, reporting to HR staff if they notice users are misusing the internet etc.

How about maintaining / updating cisco devices? None of our ASA/ Pix ever had updated firmware. Whatever came boxed is what is running. In past 5 years I never heard any major firmware upgrade.

I don’t want to jump on these guys. I just need better understanding how to make our Cisco network better .

aquilla

Hi,

Firstly congratulations on the promotion.

With regards to the duties of an engineer, there is always something to do. I'm CCNA and currently a 2nd line NOC engineer. The company I work for supports over 100 customers and over 6000 devices (mainly Cisco). My job is to work through tickets which the 1st line engineers have raised which could be about anything (customer requesting information on their estate / reconfiguring a link / router, switch or firewall changes / investigation of a fault). We also check link utilisation across our customers estate and investigate any "abnormal" usage. Our monitoring (via SNMP) can also report errors and discards on an interface (if configured), so we also check that and investigate if necessary.

Just because alarm bells aren't ringing doesn't necessarily mean everything is fine. As Petedude mentioned above, they should look at carrying out proactive checks on devices. You say you have 200 sites - that's a lot of equipment. Your engineers should be checking the logs for security breaches / link issues or switchport issues (e.g. error-dis, STP problems / duplex mismatch) / environmental issues.

Here's a quick story - we manage a MPLS network for a customer. The CE MPLS router in Italy appeared to be working fine, it was up on our monitoring, we could log into it and the site were working. Nothing appeared to be wrong. One of our engineers logged in to do a quick check and found 2 of the 3 fans in the device were not working. Not good. Incident ticket was raised and a call to Cisco TAC to have the part swapped before the device failed completely.

What is your monitoring system like? Can it be streamlined with proactive syslog alerts from the device? Do you have copies of the config for *all* the devices you manage? Nothing worse than having to replace a device and finding you don't have a recent copy of the config (esp if it's an ASA or PIX with a ton of rules). We use Network Configuration Manager from Solarwinds. Very easy to setup and it will pull down the latest config once it detects a change. We can also compare versions of the config so we can see if someone made a change. If you have the option to get it running, perhaps your engineers can look after that.

If, after all that, I have some spare time, I may indulge in a spot of studying for the CCNP exam.

aquilla

lboppi wrote: »

Thanks for the reply. Well we have Solar Wind NPM and Palo Alto Firewall to monitor traffic, syslogs, traps etc. They are monitoring these, reporting to HR staff if they notice users are misusing the internet etc.

How about maintaining / updating cisco devices? None of our ASA/ Pix ever had updated firmware. Whatever came boxed is what is running. In past 5 years I never heard any major firmware upgrade.

I don’t want to jump on these guys. I just need better understanding how to make our Cisco network better .

With regards to the IOS, what came with the box may not be the best option now. If nothing else you should ensure your estate is running the same version of code across the board, it need not be the latest.

You could get your engineers to do an audit of what is currently running on your estate and then use the tools at Cisco (Feature Navigator and Bug Check) to see which version would be best.

We try and keep a customers estate on a particular code version for stability. If an issue is discovered we then look at what options are available and plan the upgrade if necessary.

Also, what's your network documentation like? Perhaps your engineers can start preparing something (even if it's just documenting which ports are in use and what plugs into it, is a port a trunk or an access port, if an access port - what VLAN)?. They may hate doing it, but if you suffer a major failure and need to rebuild the network or understand something, it could be a lifesaver.

fredmoogie

LAN/WAN optimization
security audits on all hosts/devices
documentations
monitoring
running updates/patches on IOS

the list goes on....many things to do..never a dull moment. if those guys have too much free time, perhaps time to re-assess their functions and the needs for 2 guys.

thall860

Sorry if I am out of line on this but when I read we have two people who do nothing it raised a red flag saying why two?

If two people have nothing to do, maybe only having one would stop lowering the productivity of all the other people they go around and chat with.

lboppi

Thank you all for your feedback . Now I have a better idea what our two guys should be doing. I will be working next few days coming up a list of things to do from your suggestion. I can already see that my staff will hate me for it but I guess that comes with the job.

I will be ordering Network Configuration Manager from Solarwinds, I have seen the demo version and it is a great tool. We have APM and NPM so I know Solarwind products are good.

Thank you again for your feedback. I really appreciate it.

jibbajabba

lboppi wrote: »

Thank you all for your feedback . Now I have a better idea what our two guys should be doing. I will be working next few days coming up a list of things to do from your suggestion. I can already see that my staff will hate me for it but I guess that comes with the job.

I will be ordering Network Configuration Manager from Solarwinds, I have seen the demo version and it is a great tool. We have APM and NPM so I know Solarwind products are good.

Thank you again for your feedback. I really appreciate it.

We don't have any Cisco certified staff in our company but we do work solely with Cisco kit and we simply have "guys" dealing with it .. Our 'network guy' just recently had to do a full VLAN audit (after an acquisition) , create allow lists on trunk ports, check the spanning tree configuration (create master where applicable), sort out ACLs (man so many copy / paste errors it was unreal), remove unnecessary configuration (again, vlans / acls).

I had to sort out switches on our ISCSI VLAN - making sure the settings are approriate (jumbo frames for example) and upgrade switches accordingly (which involved adding new stacks onto existing stacks and migrate server / sans to it) ...

Documentation - that is a big thing .. I would bet my several cases of beer that no matter where you go - there is ALWAYS SOMETHING missing

Heck, we didn't even have proper documention until we were bought off so we had to do that. Which included a list of equipment including their details (IPs, firmwares, serials, TAC status) ...

Man, I could go on and on lol - and I am not even a CCNA but I am 100% certain your guys would hate me until doomsday

networker050184

Well, first off, if there are no issues they are doing something right! This isn't an easy task.

That being said, it sounds like its time for an audit. Up to date network maps and documentation. Configuration and code standardization would be next on my list. Have them start up a wiki or something similar and write up "how-to's" for common tasks. Never know when they may move on to another gig (especially if you start harassing them and making them feel threatened) and you are left without anyone knowledgeable on the network. After all that is done then its time to sit down and see if there are things that can be improved in the network.

As far as why two people, you never want to back yourself into a corner of having only one "guy" who can do something on staff. What happens if he needs to take time off or there is an emergency? Or what if he just straight up quits one day? If your budget can handle two then I'd keep two on staff.

jibbajabba

networker050184 wrote: »

Well, first off, if there are no issues they are doing something right!

If you take a switch out of the box and give it an IP, you can plugin devices and they can see eachother just fine - doesn't mean it is working right - doesn't take much to get "stuff" working

networker050184

jibbajabba wrote: »

If you take a switch out of the box and give it an IP, you can plugin devices and they can see eachother just fine - doesn't mean it is working right - doesn't take much to get "stuff" working

Yeah if it was that easy everyone would be in networking and the pay wouldn't be nearly as good.

And if you read the OP there are 200 remote sites. They don't just work on their own out of the box.

nerdydad

networker050184 wrote: »

Yeah if it was that easy everyone would be in networking and the pay wouldn't be nearly as good.

And if you read the OP there are 200 remote sites. They don't just work on their own out of the box.

I'm going to have to agree with you on this, let's face it, if you document everything when you set it up, and no new sites are coming online, there may not be much to do everyday. Let's face it, in a position like a NOC, you are there in case something breaks, now I have never been in a situation where nothing broke, but had plenty of slow nights to get some studying done.

It sounds like they hired 2 guys when everything was being set-up, and they kept them after everything was stabilized, I would hate for anyone to loose their job, but it sounds like at this point they don't need 2, however when it is time to upgrade some kit or there is rapid expansion in the company, these 2 guys would be very busy.

And no, there are not constant patches to install on Cisco gear, so forget that OS mentality, run current code, but usually not the most current, let others sort the bugs out.

Trifidw

If they reduce to only 1 person than they will not have any cover when that person is on AL/sick. I'd keep the 2 people but perhaps look at some additional duties that they could be doing.

I'm guessing you don't use VoIP/the network guys don't support the VoIP infrastructure? This takes up a large proportion of our support time at work.

Forsaken_GA

lboppi wrote: »

How about maintaining / updating cisco devices? None of our ASA/ Pix ever had updated firmware. Whatever came boxed is what is running. In past 5 years I never heard any major firmware upgrade.

Unless you actually need to upgrade the code, leave them alone. You do not touch network gear lightly, especially if everything is running fine. This is not like your operating system where you upgrade for the sake of staying current. New IOS images introduce new bugs, and it gets cute when Cisco decides to change the way features operate. If you insist that they update to more current code, make damn sure they have the equipment to test it on first, as well as a proper testing methodology, so you don't have any nasty surprises.

Other than that, figure out how their monitoring looks. Asking them for a report of link failures/flaps for the last 30 days. They should be doing capacity auditing, ask them for a capacity report for the last 3 months. How about configuration management? Are they actually doing it?

If they can provide you with this kind of stuff within a day or two, leave them the **** alone. They're keeping the network running like a well oiled machine. They're doing their job.

If they it takes a week to get you what they want.... then inspect the quality of it. If it looks good, again, leave em alone (they probably just put the solution together that week..., but if they can pull that off on short notice and make it look good, don't tread on it).

If they can't, then you've got some lazy folk who are only being reactionary. Buy them copies of The Practice of System and Network Administration, and tell them to follow it and implement the guidelines therein that are appropriate to your organization.

nerdydad

Forsaken_GA wrote: »

Unless you actually need to upgrade the code, leave them alone. You do not touch network gear lightly, especially if everything is running fine. This is not like your operating system where you upgrade for the sake of staying current. New IOS images introduce new bugs, and it gets cute when Cisco decides to change the way features operate. If you insist that they update to more current code, make damn sure they have the equipment to test it on first, as well as a proper testing methodology, so you don't have any nasty surprises.

Like NATing on the ASA 5500's?

shodown

Forsaken_GA wrote: »

Unless you actually need to upgrade the code, leave them alone. You do not touch network gear lightly, especially if everything is running fine. This is not like your operating system where you upgrade for the sake of staying current.

I have a lot of small business customers and this is one thing that irks the hell out of me. They like to upgrade there systems just to upgrade them, small things like firmware, or patch files, or COP files, and I try to explain that unless they are being hit by that bug or its a security threat, then let it go. We just don't patch things to be patching. But since they are all M$ guys they tend to think differently.

networkjutsu

Forsaken_GA wrote: »

Unless you actually need to upgrade the code, leave them alone. You do not touch network gear lightly, especially if everything is running fine. This is not like your operating system where you upgrade for the sake of staying current. New IOS images introduce new bugs, and it gets cute when Cisco decides to change the way features operate. If you insist that they update to more current code, make damn sure they have the equipment to test it on first, as well as a proper testing methodology, so you don't have any nasty surprises.

This is very true. In my previous employer, before we rolled out new IOS out to the whole network we have to "certify" it first and made sure that it was stable in a production environment for months. But, there were still unforeseen scenarios even though it has been "certified". There was a time that the IOS has been "certified" and when it was rolled out to all ~7000 routers there were hundreds that affected the WIC-1DSU-T1-V2 card. It didn't cause any circuit outages but it affected the ability of the NOC to check statistics of the WIC. There's one more IOS story but I am getting lazy to type.

Trifidw

nerdydad wrote: »

Like NATing on the ASA 5500's?

Perfect one. We usually do a software update on all network devices to bring them all inline once every 3 years but I doubt we will be able to do the ASAs unless we encounter a serious bug and will have to bring in a specialist.

Turgon

lboppi wrote: »

I recently took a position of an IT director. In the past I worked as Helpdesk technician, desktop technician, and Microsoft System Administrator and IT Project manager. Now finally I took the IT Director position few weeks ago.

As you can see my background is on Windows platform and I have very little knowledge on Cisco side.

Under my supervision we have 2 Cisco Administrators. In the past I have notice that these 2 administrator hardly do anything. They will chat online, play in face book, stop office to office and chat and do nothing. When I was their coworker I jokingly said many time “ what do you guys do all day” , and their reply was “ everything working fine, we have nothing to do, if you see a problem , let us know” .

Now that I am their boss, I want to make sure they are doing what they supposed to. From my Windows system administrator days I know even if every thing is working there are lot a things to do such as – updates, monitor servers , or start finding things to improve.

Here is our network environment. We have 200 remote locations which all connect to our central office via VPN. Each location has cisco ASA, PIX and router and in our central location we have VPN concentrator.

My question is to all CCNA – in your work what do you do every day? If everything working as it should does that mean there is nothing to do on Cisco network? How often cisco release firmware updates? How often do we need to take a look at entire network infrastructure and if we need to redesign the network.

The work of CCNA holders really varies. I recall a lovely picture on routergod in 1999 which had a squad of soldiers under fire and the caption was 'CCNAs sent in first to give CCIE's covering fire'. Nice

CCNA qualified people working in a purely networking role should be first line of defence there, and while it will be breeched, it should be strong. The CCNA does not cover load balancers, proxies, PIX, ASA and other things so please be gentle.

N2IT

lboppi wrote: »

Thank you all for your feedback . Now I have a better idea what our two guys should be doing. I will be working next few days coming up a list of things to do from your suggestion. I can already see that my staff will hate me for it but I guess that comes with the job.

I will be ordering Network Configuration Manager from Solarwinds, I have seen the demo version and it is a great tool. We have APM and NPM so I know Solarwind products are good.

Thank you again for your feedback. I really appreciate it.

Just keep it facts based and remove all emotion and you will disarm them. Good luck and great job on the promotion!

lboppi

Reading all of the comment I see there are 2 schools of thoughts. Some say “keep the system current with new patch and new ISO” other say “ if it is working then leave it as it is and don’t go in to latest and greatest” .

I do see value in both. If it is not broken then leave it alone. It has its merit. I , who has MS background my mentality is always to update/ upgrade but that may not be the case when it comes to Cisco products. That is why I opened this thread to see what others are saying.
Here is an example, one of my staff build a new virtual print server and it is working great. I just logged in to the new server and saw the spac ( I know , I got a get out of that habit and trust my guys) and I saw he build the server with 1 CPU. I know if we add another CUP it would run better, I have the resource so why not add another CPU.

My question to Cisco guys , just because it works does that mean this is the best you can do? If a ASA or 2656 XM is working with out any problem with 5 years old firmware or configuration, do we just leave the device alone or do we try to see where we can improve the connectivity, reliability etc?

Roguetadhg

My understanding with upgrading cisco products is this:
1. Upgrading may change functionality of commands used to perform critical every-day work.
2. IOS bugs that never 'bugged' you, have a real chance of bugging out.

Adding a processor isn't the same like adding a new OS.

networker050184

lboppi wrote: »

My question to Cisco guys , just because it works does that mean this is the best you can do? If a ASA or 2656 XM is working with out any problem with 5 years old firmware or configuration, do we just leave the device alone or do we try to see where we can improve the connectivity, reliability etc?

I'd tread lightly with "making improvements." While its always good to make sure things are running optimally, the world of networking often runs on the "if ain't broke don't fix it" mentality. I've seen core routers on the internet that haven't been brought down for updates in five years. I've little switches humming along for ten years.

If you really want to upgrade the infrastructure make sure its a well thought out plan. One reason to look into upgrading your infrastructure is outdated gear that has been put on end of support by the vendor. A quick google search shows the EOS for 2600XM routers is march of this year. So it is about time to start testing new products and putting together a migration plan.

As far as code updates, Cisco regularly publishes its security vulnerabilities. If you are not affected and have no need for new functionality in a later release there is no point in upgrading to new code. As everyone else has stated its usually far more trouble than help.

Good luck!

shodown

lboppi wrote: »

My question to Cisco guys , just because it works does that mean this is the best you can do? If a ASA or 2656 XM is working with out any problem with 5 years old firmware or configuration, do we just leave the device alone or do we try to see where we can improve the connectivity, reliability etc?

You have to have set out goals of where you want your network to be at. For example if you wanted to add VOIP or Video how could you do it with you current network? Is it possible or will you need serious network upgrades. As networker said check the EOS, EOL on your gear. Do you have backup plans to get hardware out to remote sites in cause of outtages, Do you have a detailed network diagram with all IP address, circuit providers, POC's and so on. While at times it may not be a lot to do. There is always plenty you can do to make your life better.

ChooseLife

networker050184 wrote: »

...it sounds like its time for an audit

Forsaken_GA wrote: »

...figure out how their monitoring looks. Asking them for a report of link failures/flaps for the last 30 days. They should be doing capacity auditing, ask them for a capacity report for the last 3 months. How about configuration management? Are they actually doing it?

These are the best advice in the thread, IMO.

It should be about a well-running network, and not about keeping people busy. So first of all you need to establish the baseline, find out the state the network is currently in - not firmware versions, but higher-level things like network reliability, redundancy, capacity planning, automated monitoring, transparency, documentation, and so on.

Since you say you have little knowledge of Cisco, the best way to do this is to contract an external/independent auditor or a consultant to review the network. If that's not possible due to budget constraints, try poking around yourself and as Forsaken_GA suggested, ask them to provide reports on various network-related metrics. Either way, this should give you an idea whether you are dealing with lazy employees and a disaster waiting to happen or real network gurus who tuned it to the state of perfection. This will also discover possible area for improvement.

Finally, regarding this:

lboppi wrote:

My question to Cisco guys , just because it works does that mean this is the best you can do? If a ASA or 2656 XM is working with out any problem with 5 years old firmware or configuration, do we just leave the device alone or do we try to see where we can improve the connectivity, reliability etc?

I believe upgrades and improvement projects should be dictated by business needs and not a desire to keep people busy. Sure there is always room for improvement, but why invest tens of thousands of dollars and countless man-hours into changing a system, if it has a proper design, experiences few issues, has low utilization and utilization growth rate?

higherho

They should replace that PIX! its End of life and the IOS on it has some vulnerabilities. If the network is setup correctly, then their should not be major changes needed unless its IOS related or security related. I recently had to harden our new ASA firewall and I had to go through 200 checks (took 8 hours to get that done).

On our current pix (the ASA will be replacing it the moment I can get some downtime), I always have up and running everyday I come into work. I of course do other things than Network stuff but I do monitor the traffic on our firewall. Such as; SYN attacks, major traffic going through, top sites visited (to many users using facebook!), seeing if random ip's are trying to get through and if so put them on the black list.

. Network guys have stuff to do, especially if you have security evolved and have a major network to worry about.

nerdydad

lboppi wrote: »

My question to Cisco guys , just because it works does that mean this is the best you can do? If a ASA or 2656 XM is working with out any problem with 5 years old firmware or configuration, do we just leave the device alone or do we try to see where we can improve the connectivity, reliability etc?

A perfect example of where this can break your network is, in certain 2600XM's (I believe the 50's), there was MPLS support in IOS 12.3, that was dropped in 12.4. Imagine your surprise when everything breaks because you wanted to run the latest and greatest. I have seen devices in a service provider core with 8 years of uptime, and I have heard about others with much more than that.

Turgon

One of things you have to lookout for is end of life and end of support notifications on IOS versions.