Configuration management tools: Puppet, Chef, CFEngine, agile, Nagios

UnixGuy

How many of you use a configuration management tool to manage servers?

I'm currently upgrading my skills because I will be job hunting soon, and I see these tools more and more appear on jobs ads. I haven't used any of these tools, unfortunately. I just want to know how did you manage to learn these tools? What's the scope of work expected from sysadmin?

I know there's a great deal of programming involved, and I still don't understand the meaning and scope of 'DevOps' yet. I just want to hear about your experience with these tools (how did you start, ..). I have a good software development background, but it seems that a lot of jobs expect you to have work experience with tools like Puppet and experience programming some modules or something. Excuse my ignorance


How and when did you start learning and using these tools? Do you think 'DevOps' is the future of system administration?

UnixGuy

I read many articles about the topic, but this is one of the best articles:

http://www.juliandunn.net/2012/01/13/chef-devops-and-the-death-of-system-administration/


your thoughts?

SteveO86

I use configuration management for Networking devices. Started using with SolarWinds NCM product a good 3 maybe 4 years ago. Make it so much easier to keep backups of configurations and standardize configurations across networks/customers, as well as make configuration changes.

So my thoughts are limited to the network side. I don't think configuration management will kill the role of administration, especially since configuration management has been around for some time already.

the_Grinch

I think many companies are making that slide for sure, but at a slow rate. I can use my place as an example. We host our entire website on Rackspace, along with sever other internal systems. They literally do just about everything involving the management and setup of these systems. We're given full access and can do what we want on them (such as installing needed software). All that being said, we still have separate duties (myself and my boss being on the support/admin side the programmers on the other side). I've helped on a couple of occasions due to the limited programming knowledge that I have. For the future, especially if Linux continues to grow, devops will be the future.

linuxlover

I believe Cacti/Nagios combo is used by a lot of people, but I've seen Zabbix required by many employers.

phoeneous

linuxlover wrote: »

I believe Cacti/Nagios combo is used by a lot of people, but I've seen Zabbix required by many employers.

+1000 for Zabbix. Best nms ever. IMHO, way better than Nagios.

pram

I think you're conflating two separate things here. Puppet/Chef/Nagios really have nothing to do with DevOps and 'agile' (lol.) They're pretty much just standard System Administration tools.

As for how you gain experience with them, what exactly is preventing you now? Setting up a puppet master is almost trivial, they even provide a VM for practicing with:

Puppet Labs | Download the Learning Puppet VM

Nagios can be kind of complicated, and definitely requires a bit of programming knowhow. Again though, theres a lot of material and monitoring plugins available to reference:

Nagios Exchange - Plugins

EDIT: I think Nagios may be harder to learn on your own though, simply because you'd have less to monitor than in a real environment.

UnixGuy

pram wrote: »

...

As for how you gain experience with them, what exactly is preventing you now? Setting up a puppet master is almost trivial, they even provide a VM for practicing with:

Puppet Labs | Download the Learning Puppet VM

.....

That's what I was looking for, thanks! So you think a good start would installing a Puppet master server VM and play with it. I have some Red Hat and Solaris VMs, so this should be ok?

UnixGuy

SteveO86 wrote: »

..

So my thoughts are limited to the network side. I don't think configuration management will kill the role of administration, especially since configuration management has been around for some time already.

Interesting. Although I can see a shift of the mindset of a sysadmin, and I can see the Sysadmin career shrinking with more and more businesses shifting towards the cloud. True cloud provider will still need admins, but again one of the goals of cloud computing is to reduce system administration overhead.

dontstop

Configurations management is a win, Puppet has been moving into other areas like networking with support for Juniper and Cisco becoming stronger by the day. Ensure that you logically build your manifests and store everything in an RC (revision control) system like SVN, GIT or other. Well worth your time, If you use it correctly.

UnixGuy

the_Grinch wrote: »

I think many companies are making that slide for sure, but at a slow rate. I can use my place as an example. We host our entire website on Rackspace, along with sever other internal systems. .... All that being said, we still have separate duties (myself and my boss being on the support/admin side the programmers on the other side)...

So with Rackspace you don't really need to do any system administration? who takes care of backup? who applies Information security policy? who monitors the server? who takes care of storage?

the_Grinch wrote: »

For the future, especially if Linux continues to grow, devops will be the future.

I think it is growing rapidly!

UnixGuy

linuxlover wrote: »

I believe Cacti/Nagios combo is used by a lot of people, but I've seen Zabbix required by many employers.

Interesting, I never heard of that one before

UnixGuy

dontstop wrote: »

Configurations management is a win, Puppet has been moving into other areas like networking with support for Juniper and Cisco becoming stronger by the day. Ensure that you logically build your manifests and store everything in an RC (revision control) system like SVN, GIT or other. Well worth your time, If you use it correctly.

Great points. What do you think is the best way to start with Puppet? What are the essential things to learn on my own?

pram

UnixGuy wrote: »

That's what I was looking for, thanks! So you think a good start would installing a Puppet master server VM and play with it. I have some Red Hat and Solaris VMs, so this should be ok?

Yes. I'd start with something simple like managing sudoers/resolv.conf. Once you understand the basic structure of a module and you've made a few manifests it should be easy to grasp. The language is declarative, so its not programming as we traditionally understand it. Your puppet manifest is describing how the server should be, rather than the steps to get that way.

UnixGuy wrote: »

So with Rackspace you don't really need to do any system administration? who takes care of backup? who applies Information security policy? who monitors the server? who takes care of storage?

Rackspace operates much like a shared hosting service, you call their Administrators for support. Contrast this with Amazon's model, which is essentially unmanaged. Rackspace is kind of like a hybrid host/managed service provider.

pram

Heres an example of a manifest I get a lot of use from:

class whatever {
file { "/tmp/thing.tar":
owner => "root",
group => "root",
mode => 0440,
source => "puppet://$puppetserver/modules/whatever/thing.tar",
}
exec { "tar -xvf /tmp/thing.tar":
cwd => "/tmp",
creates => "/tmp/thingy",
path => ["/bin", "/usr/bin"]
}
}


This makes class whatever, which can be applied to specific hosts with nodes.pp with "include whatever"

What this does is tells the client that there should be a file called thing.tar in /tmp, and that this file comes from the puppet master. The file itself will be located in your modules directory, under files. So the puppet agent will automatically handle downloading it and put it into /tmp

The second part tells it to exec tar on the file, and it only does this if /tmp/thingy doesn't exist (thingy would be something inside the tar) this is how we prevent it from running over and over again.

Hopefully this helps explain what I mean by 'describe how the server should be'

phoeneous

UnixGuy wrote: »

Interesting, I never heard of that one before

It's amazing what you can do with Zabbix.

the_Grinch

As part of our services with Rackspace they are performing backups for us. They do all the monitoring of the servers, but honestly we've not had one second of downtime in the year I have been there. All that being said we have full access to our servers and can do what we like with them. They handle all the security as well and of course I wondered if they really were doing anything, but found out last week they were. Our IT Manager called me over saying something was wrong with the website. I checked and it was loading just fine for me. I walk over to his machine and he couldn't view the website nor log back into the servers. He gets an email just as I got there saying they detected what appeared to be an SQL Injection (and based off of the work he was doing I could see that) and blocked his IP address.

We do our own account creation within the server farm we have and given how we use it we don't really have any security policies (beyond updates, password complexity/changing, and the like). Off hand I believe they'd do this for us if we wanted, but we handle it on our own and only contact them when there are issues.

UnixGuy

pram wrote: »

..


Hopefully this helps explain what I mean by 'describe how the server should be'

Thanks man, that's very helpful I hope in my next job I'll get to play with some of these tools. I'm currently updating my skills, so I'll puppet to my list of things to learn

UnixGuy

phoeneous wrote: »

It's amazing what you can do with Zabbix.

What are the advantages of using Zabbix? What's the ideal environment where you see Zabbix as the best solution? What's the best thing about Zabbix from your experience?

Examples would help me understand

UnixGuy

the_Grinch wrote: »

As part of our services with Rackspace they are performing backups for us. They do all the monitoring of the servers, but honestly we've not had one second of downtime in the year I have been there. .....

Pretty scary because this mean that there's absolutely no need for a qualified System Administrator ? Do you think such solutions will affect the availability of traditional System Administration jobs?

petedude

UnixGuy wrote: »

How and when did you start learning and using these tools? Do you think 'DevOps' is the future of system administration?

I'll answer the second question first: In large Linux server environments, we're pretty much there. Can't tell you how many postings I've heard of lately where people are looking for Puppet or Chef. Don't see as much CFEngine or Nagios, but I do see them. I think Nagios is eventually going to have sizable market shares in companies that don't want to/can't afford to buy MS/IBM/CA network management software.

Agile: that's a programming methodology. It's been here quite a while. Various theories have been put forth for its popularity, but basically I think the Agile revolution amounts to this: It's almost an end-user revolution against the stiff, formalized black-box development methods of times past.

As far as learning any of this stuff, personally. . . I've just started learning what it it all IS. I was toying with getting certified on Puppet and/or Chef briefly, but I had to get my brain back on track as there are other foundations I would benefit from building.

phoeneous

UnixGuy wrote: »

What are the advantages of using Zabbix? What's the ideal environment where you see Zabbix as the best solution? What's the best thing about Zabbix from your experience?

Examples would help me understand

Advantages? Well, it'a a network monitoring system that goes beyond just 'host up/down' checks. Basically it uses metric checks called items which can monitor anything from the temperature on a UPS, performance counters on servers, interface throughput on network devices, webpages, the list is endless. At the time of this writing, I currently have about 2000 items actively polling my network. Probably easier if you just read about items than me explaining.

https://www.zabbix.com/documentation/1.8/manual/config/items

It's suitable for all types of environments where you want to truly monitor your infrastructure. 10 devices, or 10,000 devices, it will still do the same thing. Before I was using Zabbix I thought everything was running smoothly... boy was I wrong.

In my experience, there are two great things about Zabbix.

1) It's FREE. No licensing to worry about.
2) You can monitor (and graph!) anything that you can think of.

Homepage of Zabbix :: An Enterprise-Class Open Source Distributed Monitoring Solution

dontstop

UnixGuy wrote: »

Great points. What do you think is the best way to start with Puppet? What are the essential things to learn on my own?

Start at home! Build a Server or *two* (personally, learn how to make them redundant now as most people leave that to later and it never happens [be it as simple as having a cloned server, load balanced machines, or something more exotic]) and start managing basic services on your home machines.

* resolv.conf
* NTP Servers
* motd/motd.txt
* Default installed apps (openssh, mtr, iptraf, atop)
* etc etc.

Start small, picking more and more things you want to control under puppet. Remember, you don't need to control everything from day 1.

Another cool project you can work on is trying to build a puppet manifest that builds a puppet server. So, if your puppet server was to die how could you quickly with only your backed up manifests and a fresh install have your server quickly rebuild itself as a puppet master server

Have fun & experiment.

Edit: the most important thing is learning to make things reusable and modular.