Linux patching best practices

srj

Does anyone have any good resources for Linux patching best practices? Most of our Linux systems are RHEL/Cent. I am a newer Sys Admin and was recently tasked with getting our Linux environment patched. We have a Spacewalk server to work with.

My biggest concern is determining whether a patch should be applied or not. Our environment is currently out of date by a few point releases per system. It is difficult/impossible to consider every possible update when the systems are so out of date. I would spend all day reading release notes. We're looking at 120-250+ patches per system.

For Linux admins, do you typically apply all updates that are available? Is it common for you to not update certain packages? I understand that each patch does run some risk of breaking things. Do you typically just apply errata updates? I have been trying to contact the "owners" of the systems. Many of them are software developers, so I try to get information about any packages that they are aware of that might break their environment. We do have a number of less savvy individuals who are admins of an application, but don't have much of an idea of any interactions with patches.

My thoughts right now are that worst case scenario is I revert to the pre-patching snapshot. I could also use yum to downgrade specific packages that I believe might be problematic.

Thank you all ahead of time. This seems to be one of the few areas that I cannot find much information.

Verities

Best practice is have a test lab where you could have a few boxes that host all the applications your devs work with and applying patches to the boxes. You would then have them test the functionality of their applications on those boxes, before rolling out to your production servers. If you don't have that kind of environment, another way would be to update maybe 1 server at a time that is hosting an application, letting it cook for a few days and have the users report back any issues. You definitely want to have a back-out strategy if things go south.

In our environment, we typically have to update to the latest round of patches after our engineering department has tested them, we'll apply them to 1-2 production servers and then let them cook a few days to monitor for any issues. This is usually about a month long process from start to finish. Implementation usually takes the longest since we have so many servers and if a server is going to fail, it usually does during patches.

JockVSJock

Some good advise.

The group that I work for is also behind on patching too...

Tearstone

When I first started patching our environment, I wanted an approach which management would understand. I would use the second Tuesday of the month (Microsoft Patch Tuesday), to snapshot the patches that were available. I would apply all patches to my test boxes which mirrored production environments. I created reports and provide to both management and information security. The third Tuesday of the month, after the patches had a chance to cook and simmer down over a week, I would roll the patches out to all production systems.

If you want to limit the amount of patches that get rolled out, I would recommend just focusing on security patches. I chose to roll all available patches out to systems, with Red Hat you pay top dollar for them to vet these patches out before you bring them into your environment. Maintaining your security patches is apart of an important foundational approach at minimizing security risks.

JockVSJock

The previous admin had two test boxes (32-bit an 64-bit) and he tested them from here.

Once they were good, he deployed them to the rest of the prod environment.

I need to get into the habit of reading the Red Hat releases.

Tearstone

JockVSJock wrote: »

I need to get into the habit of reading the Red Hat releases.

That is a luxury that some admins do not have in an environment that is always behind the 8-ball as it is as we continue to be in positions of doing the jobs of what used to take two-three people to do well.

Verities

Agreed, with the above, security patches should be your highest priority. Any other updates should be lower on the list.

Check this out for some additional reading:

http://www.sans.org/reading-room/whitepapers/bestprac/practical-methodology-implementing-patch-management-process-1206

I've never used Spacewalk, but since its the open source version of Sattelite, I would imagine the process being similar:

http://www.redhat.com/f/pdf/rhn/WHP0008US_RHN.pdf

Tearstone

Verities wrote: »

I've never used Spacewalk, but since its the open source version of Sattelite, I would imagine the process being similar:

http://www.redhat.com/f/pdf/rhn/WHP0008US_RHN.pdf

This is what I used to freeze the patch levels on all systems. I just ran a satellite-sync on that second Tuesday of the month. Any system in the organization registered with the Satellite would all get the same patch level.

Before I had Satellite, I had a somewhat elaborate way of running my yum updates with a bunch of exclusions of patches that were populated in the RHN over the course of that week.

W Stewart

I would just stay up on security patches and bug fixes. There's no need to update multiple packages on a system running a critical application if there's nothing wrong with the version of the package that you're using. If we were to go by that logic then everybody should be using a bleeding edge distro like fedora or arch just so they have the newest stuff. I'd just keep up with security notices for whatever distros you're running and production because ultimately, your biggest concern should be having one of your servers compromised.

JockVSJock

Verities wrote: »

Agreed, with the above, security patches should be your highest priority. Any other updates should be lower on the list.

Check this out for some additional reading:

http://www.sans.org/reading-room/whitepapers/bestprac/practical-methodology-implementing-patch-management-process-1206

I've never used Spacewalk, but since its the open source version of Sattelite, I would imagine the process being similar:

http://www.redhat.com/f/pdf/rhn/WHP0008US_RHN.pdf

Thanks for the SANS whitepaper.

Is it me or do most of those Whitepapers outdated? That one on patch management if from 2003. I read another one from there the other day on Tripwire that was also from 2003 too.