How do you guys handle Linux Updates/Patches?

DoubleNNs

Do you use scripts? Configuration Management? Satellite/Spacewalk? Or do you do you practice immutable infrastructure, and simply replace old instances with new ones that have updates pre-baked (update during provisioning)?
I also see the likes of Katello and RH CloudForms System Engine from a Google search.
On top of that, what is your methodology of determining what gets updated and what doesn't?

thomas_

I'm not really a sysadmin, so you probably don't really care about my answer. I run a few websites using DigitalOcean droplets with CentOS. I don't do any of what you mentioned. Every once in awhile I will SSH in and do a "yum makecache" and then a "yum update" and just update all of the packages. The blogs are low traffic and don't get a lot of visitors to them, so I'm not too concerned with them breaking. Maybe when they are higher traffic and make more money I'll look at doing those things. I havent had anything break yet(that I know about). I would rather mitigate against known issues than worry about potential unknown ones caused by the new updates.

Mike7

I automated patching of my CentOS VMs via yum-cron. You can configure it to download patches only or apply patches after downloading.
The VMs are backup daily, so I can roll-back if necessary. Works well for me so far.

brombulec

Configuration management = Puppet/chef/ansible
Patch management = satellite/katello/spacewalk

End of story

DoubleNNs

@Mike7
I didn't know yum-cron existed until now. Do you think you could point me to good resources about it?

I'm particularly interested in what benefits yum-cron has over "manually" managing yum via a regular cronjob. Also how does the download only mechanism work? And if I download-only patches on say 6/15 can I then apply only the previously downloaded patches (without downloading new ones) on 6/30?


@brombulec
Where's the SaltStack love?

If an environment is mostly RHEL (as opposed to CentOS or SuSE/Debian-based) do you recommend Satellite over Spacewalk? Additionally it seems like Katello and CloudForms have tons of features not in Satellite or Spacewalk. Do you recommend those?

Even more important, what benefits does having one of those Patch Management systems provide as opposed to simply scheduling yum or Config Mgmt to update?


Sorry for the barrage of questions, I've just never really thought too much about package management, until now.

brombulec

@DoubleNNs:
Of course you can use yum-cron and other tools but:
1. With Satellite/spacewalk you have a better control. You can control what, where and when will be installed.
2. The reporting included in Sat/SW shows you the status of all your systems
3. You can use it for a ConfigMgmt (somehow limited but still useful) - I prefer to use Puppet for these tasks. Puppet is a tool designed for config management and automation and I see no reason to avoid this great tool in the environment. Chef is a little bit complicated but also useful.
I'm not familiar with SaltStack.

Xavor

Saw your post on reddit, but answering here:

We use spacewalk and puppet. I've only casually looked at SaltStack, Ansible, and Chef and know how to talk about them if they come up in discussion.

You asked about cloudforms and satellite vs. spacewalk. If you can get your customer/employer to use the official tools I would recommend that method because you have some measure of support you can call if everything goes sideways. Especially if you are running a RHEL-centric shop.

Using a tool like spacewalk (or Red Hat's branded satellite 5) does more than your typical yum update. Specifically, it is a centralized historical record of what happened to a server, offers kickstart functionality, and you can manage non-standard packages as a repository for servers. It's also good for those of us whose servers are not able to pull updates directly from the internet and have to manage an air-gapped network.

Satellite 6' equivalent is Foreman which integrates Puppet, Pulp, and others. I have limited exposure to this.

A lot of these decisions can be driven by the requirements of the site you are maintaining. If it's 100 servers, spacewalk/satellite 5 could be the best option where you make changes in the gui and push it out to the systems on a schedule. If it's 1000 servers, you may consider doing satellite 6 because of the puppet integration or just roll your own puppet master + something else.

Separate questions for you: Do you have a build environment that uses something like git where you update code, and a continuous integration build server re-rolls your AWS servers from AMI's? Does the AMI have code to pull in configs from the git repo to rebuild and scale a web app automagically?

DoubleNNs

Honestly, this question was spawned from the fact that I'm contemplating taking on a new role in a new company. I don't know what their environment is like, just yet. No AWS however. They had tried AWS in the past, didn't like it, and are planning on building their own internal, on-prem cloud. I spoke with 5 people on the team, and each of them mentioned Linux updates in some form or another, so that's definitely one of the earliest projects I'd have to tackle.

Salt is essentially a Python version of Puppet/Chef. Whereas Ansible is also Python-based, Salt uses an agent by default (but allows SSH connectivity as well) which is much faster than SSH, as well as includes a bunch of extra features compared to Ansible.

In the past I used Git to version control all the Salt configuration management tools and provision new EC2 instances using Salt. Instead of creating a new AMI, we'd have salt just create a bare-minimum AMI and immediately add the config on top of it. Go get a coffee and the new server was fully stood up before you could make it back. We didn't deal much with scale or even clusters -- in that environment we had a bunch of "special snowflake" servers, with human readable names (`ssh spiderman`) -- nor did we use a CI server, altho I had suggested it a few times.

Mike7

DoubleNNs wrote: »

@Mike7
I didn't know yum-cron existed until now. Do you think you could point me to good resources about it?

I'm particularly interested in what benefits yum-cron has over "manually" managing yum via a regular cronjob. Also how does the download only mechanism work? And if I download-only patches on say 6/15 can I then apply only the previously downloaded patches (without downloading new ones) on 6/30?

When you do a manual yum update, it needs to download all the patches and this may take a while.
yum-cron can download patches first, and if you can configure it to install them automatically upon download
I guess the main benefit is that it is built-in, is stable and RedHat supports it.

yum install yum-cron to install and edit /etc/yum/yum-cron.conf
Some yum-cron.conf options below. More from https://www.certdepot.net/rhel7-configure-automatic-updates/

• update_cmd = value specifies the category of upgrade where value can take:
  • default for yum upgrade,
  • security for yum –security upgrade,
  • security-severity:Critical for yum –sec-severity=Critical upgrade,
  • minimal for yum –bugfix upgrade-minimal,
  • minimal-security for yum –security upgrade-minimal,
  • minimal-security-severity:Critical for yum –sec-severity=Critical upgrade-minimal.
• download_updates = yes/no specifies whether these available updates need to be downloaded.
• apply_updates = yes/no defines whether these available updates need to be applied

Think of it as RHEL's equivalent of Microsoft Windows Updates. My VMs were running 24 x 7 and I was just too lazy to log in regularly to apply patches. Depends on your business needs, yum-cron more be just right or just too simple... er primitive.

DoubleNNs

With `apt-get` I can run an `update` on 6/15, which will tell the server what packages and version numbers are available. If I then run an `apt-get ugrade` on 6/30 it will only be able to install/upgrade packages that it was made aware of on 6/15.

This provides the benefit of staggering updates. I could have an application cluster of 10 servers. Run the `apt-get update` on Monday morning and then run an `apt-get upgrade` on 2 servers for every day of that week. Because the servers all downloaded the package index/cache the same day, I can make sure that the servers updated on Friday don't get patches created on Wednesday; doing this would cause the application cluster to drift in similarity.

I'm trying to figure out if yum-cron has this ability, and what benefits the other services (katello/spacewalk/satellite/foreman/CloudForms, etc) provide. Additionally, even tho I've worked on RHEL and CentOS before, I've never handled RHEL package management. I'm not too sure how the Red Hat Network and server subscription play into any of the aforementioned tools.

I'll probably have some time to research the tools a bit more late this week, and maybe stand-up 1 of the services at home next weekend. But at the moment, I'm still struggling to discover (and articulate) what features to even look for in a Linux (probably more RHEL-centric) patching solution.

JockVSJock

Here is how I manage my patches, in my mind, nothing Earth shattering here:

- Operate discounted Red Hat Satellite, therefore have to download ISOs weekly/monthly, run md5 checks against each
- Have test systems and install patches here first, see how the systems react and have developers test
- If all ok, install on production systems. Always create a snapshot first in vCenter and once successful, delete the snapshots in 24/48 hours

- Have cron jobs that run daily that check into RH Satellite:

yum clean all ; yum repolist ; yum check-update 

Would like to get into Puppet, Chef or Anisble. I would love to get VMWare Tools to be pushed successfully from Satellite or another solution, instead of my scripting things up.

JockVSJock

Forgot to mention.

There are a number of free pdfs that deal with Linux and patch management. I found one from Michael Jang, and its around 300 pages long. Haven't read it from start to finish, just skimmed it...

Mike7

JockVSJock wrote: »

I would love to get VMWare Tools to be pushed successfully from Satellite or another solution, instead of my scripting things up.

Have you tried open-vm-tools (OVT)?

Verities

@DoubleNNs: I use Satellite 5.7 and it works very well for patch management. The servers check in every 4 hours for updates and if there is new errata, its applied to the system. I really like the running remote commands/scripts to all my servers from a central management point however you can't create a reoccurring scheduled task.

I've currently deployed Satellite 6.1.8 and its had quite a few bugs that I didn't care for. I think the idea of having all the capabilities are great, but its really clunky and difficult to manage. You can however create scheduled tasks which is nice, but I've yet to use it since I got dragged away for another project.

Its interesting to see that Sat 5.7 got its lifetime extended to 2018 as they announced the Sat 6.2 beta and the removal of elastic search from the stack. I'm very eager to try Ansible Tower and see if its worth using. The pricing can be kind of high for small businesses, but I think the benefits of it and its agentless functionality are really enticing.

The idea of running cron jobs for updating your servers is ok if you're doing one or two, but that's not a good idea to implement in a larger environment. When considering patch management, you should look at creating a patch cycle, how to test patches, and then deploy them. Also, its recommended to use open-vm-tools over vmware tools since its very likely the vmtools stack will break after a kernel update and have to be reinstalled.

thomas_

I believe when you do a "yum update" it gives you a dialog asking if you want to install the updates. You then have the option of typing "y" "n" or "d". If you thpe "d" and press enter it will download the updated packages, but not install them.

Xavor

DoubleNNs wrote: »

Salt is essentially a Python version of Puppet/Chef. Whereas Ansible is also Python-based, Salt uses an agent by default (but allows SSH connectivity as well) which is much faster than SSH, as well as includes a bunch of extra features compared to Ansible..

Yup, totally understand. I guess I should have said I'm not proficient in them.

DoubleNNs wrote: »

In the past I used Git to version control all the Salt configuration management tools and provision new EC2 instances using Salt. Instead of creating a new AMI, we'd have salt just create a bare-minimum AMI and immediately add the config on top of it.

We do the same, but with Puppet.

I'm sure that if you are joining a team, you'll have to figure out the politics of how they are doing things as well as suggesting the patch changes for their infrastructure. Do you know the scale of the infrastructure yet?

I'm currently getting killed over the special snowflakes at my site. Puppet is great, but when you have to repeatedly spend 4 hours tweaking Puppet for one-offs you start to feel like the end doesn't justify the means.

brombulec

thomas_ wrote: »

I believe when you do a "yum update" it gives you a dialog asking if you want to install the updates. You then have the option of typing "y" "n" or "d". If you thpe "d" and press enter it will download the updated packages, but not install them.

Or you can use yum --downloadonly

DoubleNNs

I understand that you can download-only updates. How can you later install those updates, w/o downloading new versions? For some reason, I can't find that info anywhere.

The scale of the infra would be in the hundreds. More than half the servers are WIN, but their Linux footprint is steadily rising.

Verities

DoubleNNs wrote: »

I understand that you can download-only updates. How can you later install those updates, w/o downloading new versions? For some reason, I can't find that info anywhere.

The scale of the infra would be in the hundreds. More than half the servers are WIN, but their Linux footprint is steadily rising.

Well since you download the packages only, they'll be placed in the directory you specify. You can then use either rpm -ivh /absolutepathtofilename/ or yum localinstall /absolutepathtofilename/

brombulec

Or you can create your own repo and use it.
You can also create script, mount the directory with updates via nfs/sshfs, copy updates via rsync/lsync... sky is the limit

alias454

JockVSJock wrote: »

Here is how I manage my patches, in my mind, nothing Earth shattering here:
Would like to get into Puppet, Chef or Anisble. I would love to get VMWare Tools to be pushed successfully from Satellite or another solution, instead of my scripting things up.

I created a SaltStack Formula for doing an automated install/upgrade of the VMware tools. basically you define what version of VMware tools you want deployed and salt deploys from the provided tarball. It also figures out if you are running rhel/cent os 6 or 7 and acts accordingly. VMware actually recommends using the open-vmware-tools packages now, and is available from the main package repos for rhel/cent os based 7 clients.

https://github.com/saltstack-formulas/vmware-tools-formula

As far as Satellite/Spacewalk/Ketello etc. I found them to be too much tool for my simple needs. Ketello builds in foreman, and puppet as well as repo management. I opted for building an internal yum repo server that is managed using reposync and lftp for syncing down packages monthly. I then deploy to a test base prior to rolling out to production. At some point setting up vagrant to spin up servers just for patch testing would be cool but for now is overkill since I am not in a devops environment.

Having internal repo servers also has a side benefit of being able to add any repo/package I want to our local servers. It helps to manage what package versions get deployed, keeping the infrastructure the same across the board. The other thing I do is at the end of my reposync run is run ls to **** the package names into a file, which then gets synced to a local git server. I can then look back over every month to see what packages changed from month to month.

I wrote up a blog post about setting up your own repo server Setup internal yum repositories for CentOS and RedHat Servers Part 1 | Tech notes