Inherited a disaster of an IT department

I've just started a new job and inherited an IT department of one (me). The previous tech who is being replaced by me has completely destroyed the server room. He is able to do the job, but there is no documentation with any configurations, no means or method to how he seems to do things. Installation programs are spread amongst different servers, etc. It's obvious what I have to do (beside getting rid of him), but has any one ever seen a set of guidelines or steps on organizing an IT department? If any one could provide some feed back it would be appreciated. Thanks.

Find more posts tagged with

Save $250 on 2025 certification boot camps from Infosec!

Book now with code EOY2025

Button

Comments

EV42TMAN

Well first I'd get a handle on what you have. You could use a program like nmap to start building up a simple visio diagram and document everything. Then I'd look into how in theory the network suppose to be set up compared to how it is set up. Then start fixing stuff, and if there is some thing that can be upgraded then follow change management and upgrade it if you can.

If you want more structured approach you can look for network audit forms and follow them to a tee.

JerryL

Thanks for the reply. Can't believe the mess. The guy doesn't even have standardized email addresses or account logins. Definitely will have my work cut out for me. Good idea on nmap and the audit forms. I was looking for IT organization check lists and didn't find much. Found quite a few links for network audit forms though. There's a lot of utilities and ideas under both solarwinds and spiceworks also.

QHalo

Yeah I'd start with the lower layers first. If the infrastructure isn't working properly the rest don't really matter. That's how my boss and I are working it here. Same situation. Gotta get that solid foundation. Work on the stuff that keeps you up at night.

MrAgent

If youre using Active Directory you can use this tool to map everything as well.

Download Microsoft Active Directory Topology Diagrammer from Official Microsoft Download Center

I've had to use it a few times. Itll map everything, and put it into a nice visio diagram as well.

Mishra

This thing is amazing:

Microsoft Assessment and Planning Toolkit

mrmt

JerryL wrote: »

I've just started a new job and inherited an IT department of one (me). The previous tech who is being replaced by me has completely destroyed the server room. He is able to do the job, but there is no documentation with any configurations, no means or method to how he seems to do things. Installation programs are spread amongst different servers, etc. It's obvious what I have to do (beside getting rid of him), but has any one ever seen a set of guidelines or steps on organizing an IT department? If any one could provide some feed back it would be appreciated. Thanks.

10 years ago, I inherited a messy IT department as well from one IT guy with no documents on anything, not even admin password. It took me almost 3-4 years to have a sense of owning my own server room. I suggestion is not to touch anything that is not broken. If anything broken and there is no document, I suggest that you create your own that works.

--chris--

I am ashamed to admit it, but that guy is me where I currently work. Although, on a smaller scale since anything beyond desktop support is escalated to our IT service provider (off site). We all have to start somewhere you know?

Ill be following along and working to correct this.

elTorito

I'd start by assessing WHAT you actually have in the datacenter. Use a spreadsheet (preferably on a laptop, so you can carry it around while you're in the server room) and categorize your assets by servers, switches, routers, firewalls, storage arrays and so on.

Next, find out the admin/root logins for the stuff you've found in the previous step, and test the logins. You definitely want to be able to login as administrator should something break in the next few days. Document the credentials in a free password management tool such as KeePass. You might want to keep track of which credentials are potentially insecure, and change them as needed.

Once you've finished the previous two steps, it's time to find out the WHY of each piece of equipment in the server room (and beyond). Start at the basics. Which device/server is functioning as DHCP server? Which server hosts DNS? Is there a directory service? Is there a layer 3 switch in there that does the routing? Which subnets and VLANs are in use and why? Are there any recent backups of the critical systems?

Finally, draw up a logical and physical network diagram and integrate the findings from the previous steps in there. At this point you should have a decent understanding of how things are tied together at the moment, and from there you can think about how to clean up the mess. Be careful and methodical about this. Devise a clear strategy of how you're going to approach each problem, and try to use this strategy as a guideline. Everyone can jump in and start fixing stuff left and right, but consistency is what makes a good sysadmin.

Most importantly: document everything you do right from the start! This is going to save you a lot of hassle down the road.

Iristheangel

Sadly, this is the state of a lot of companies. Sometimes it's a small oversight and sometimes it's just a consistent pattern of laziness.

Earlier this year, we did a core collapse at our data center and installed some new ASRs, IPS, etc. We did everything right in regards to creating a project and test plans, detailing the steps, copied updated configs, etc but, as I discovered recently, one big piece of the followup work wasn't completed. One essential thing was missed: An updated datacenter diagram. I ended up spending a few hours at the datacenter tracing wires, labeling equipment, and gathering IPs. I updated the old diagram with all the changes including ports and IPs and distributed it to everyone on the team. A week later, a situation happened that absolutely required the network diagram in order to troubleshoot and it saved our collective butts.

Anyways, documenting everything is definitely not the most exciting part of the job but it has the potential of saving your butt in the future. As everyone has already stated in this thread: Grab your tools and start documenting. Good luck

JerryL

Thanks for the replies.
Some definitely good suggestions and they're allappreciated. At least I have the guy here that's leaving in a shortwhile. He's using non-standard applications for email (mdaemon) andcitrix server that's designed for VM systems to connect to other PCs in aremote office, whereas a VPN tunnel and an app such as Logmein would work justfine. I'll definitely be going over everything with a fine-toothed combto learn what I can on this network.
I don't plan on touching anything except to get the specsand document what I have until I have a good handle on it. Going in with the right attitude at least butit can be a daunting task to get this organized. If I do, it’ll be a big feather in my cap.
Stay tuned…

RouteMyPacket

JerryL wrote: »

I've just started a new job and inherited an IT department of one (me). The previous tech who is being replaced by me has completely destroyed the server room. He is able to do the job, but there is no documentation with any configurations, no means or method to how he seems to do things. Installation programs are spread amongst different servers, etc. It's obvious what I have to do (beside getting rid of him), but has any one ever seen a set of guidelines or steps on organizing an IT department? If any one could provide some feed back it would be appreciated. Thanks.

Welcome to IT!

The only advice I can give you is to NOT be like 85% of IT "Pros". IT attracts some of the laziest idiots you will ever come across.

Make changes where you can but start coming up with a gameplan on bringing the infrastructure up to snuff. A good rule of thumb when entering an environment or taking it over

1. Backups? What are we doing for backups? What is our retention policy?
2. Where is the Exchange? Where is the file server? Printers? What IP subnets do we have?
3. Start documenting thing as they are today

Once you have a grip on those three items, you should be good to go.

olaHalo

MrAgent wrote: »

If youre using Active Directory you can use this tool to map everything as well.

Download Microsoft Active Directory Topology Diagrammer from Official Microsoft Download Center

I've had to use it a few times. Itll map everything, and put it into a nice visio diagram as well.

Wonderful tool thanks.

iBrokeIT

"Difficulties mastered are opportunities won" - Winston Churchill

Psyco32

SANS also has some guidelines/checklists and tools that maybe able to help you: Audit Community Links

wweboy

Personally I'd love to inherit that kind of problem. You get to pick apart the mess and put it back together properly like a puzzle. Challenges are what make IT awesome. Anyway I'm getting off topic.

RouteMyPacket is correct start with backups before anything else is started you should have a good backup scheme and a full backup so if god forbid something happens you've covered your real. Others have already pointed good start points regarding getting equipment and important password information but everyone seems to forget about talking to the end users themselves.

Ask around the office about what they have to work with on a day to day basis, find out where items are stored (locally or on the network) because at the end of they day your working to make them more productive at their job. The fact that naming conventions and e-mail addresses don't jive is a small thing if its working right now leave it a long and after all the big fires are put out and your infrastructure is stable then start worrying about the little things that yes would make your life easier of course but in the big picture doesn't make a difference at this point.

I wish you all the luck and hope you can turn the network into a lean mean efficient machine I look forward to hearing about your progress as time goes on. Don't get frustrated you've got a great opportunity on your hands to really learn a ton of stuff and a mass a wealth of usable skills.

JerryL

Wow!!

I'm both surprised and impressed by the amount of people who have replied to this post and all with very good ideas. Thanks...

One of the first things that i'm going to be doing after the guy leaves that i'm replacing is to take a physical inventory of everything. I've already started to talk to my boss about bringing things back into conformity. Backups are done by a shareware program with zero support. No off-site backups. Explained the worst case scenario that if the building burns down; equipment can be replaced but data can't. Once i get the physical and logical topology done, i plan on going through each system to record the configurations. At least if something gets changed I'll have the way it was setup to begin with. Kind of a baseline to start with. Main thing is to break it down into bite-sized chunks. Otherwise it can become overwhelming.

The only bad thing about a situation such as this is that if something happens to an application or something else that you don't know yet, it can become daunting.

Good point on talking to the end users on getting their input and wish list. I'm very much in favour of documentation. I've spent most of my working career in the military and in either government or large company positions where documentation is a requirement. Miss not having it. This is definitely not an overnight endeavour but I will definitely be smiling and feeling great as i see steps being accomplished.

Lots of fun ahead!! LOL!!!!

Thanks guys.