RHCE todo list and other stuff
Xavor
Member Posts: 161
Busy at work so I don't check this forum as often as I used to. But to reply to a pm from brownwrap this was the link to a post buried somewhere: http://www.techexams.net/forums/linux/107115-lpic-1-2-precursor-rhcsa-security.html
I didn't create the list though, I saw it on reddit. It shows up there as a "what should I learn for linux admins". Figured I'd move my buried reply to a new post. This is where I am on the list in italics, and how I would make some changes to it.
Note: Disable the firewall / SELinux to test your stuff works, but turn it on when you're done. You'll learn more that way. Ensure your services survive reboots. Take snapshots of your vms (unless they're Active Directory controllers...don't ask). I hate how guides online tell you to disable the security. You're lucky if all the flags they told you are correct also.
1) Set up a KVM hypervisor.
Yup
2) Inside of that KVM hypervisor, install a Spacewalk server. Use CentOS 6 as the distro for all work below. (For bonus points, set up errata importation on the CentOS channels, so you can properly see security update advisory information.)
I read into this and ended up installing a foreman server which appears to be replacing Spacewalk. It integrates Puppet and some of the functionality of Spacewalk to do provisioning and change management. It's similar to other cloud provisioning tools like OpenStack.
I'd recommend instead to build a vm and setup your own local mirror of the CentOS 6.6 repo, the epel repo, and configure a cron job to rysnc them weekly. I did that a few months ago, and wished I'd done it earlier.
3) Create a VM to provide named and dhcpd service to your entire environment. Set up the dhcp daemon to use the Spacewalk server as the pxeboot machine (thus allowing you to use Cobbler to do unattended OS installs). Make sure that every forward zone you create has a reverse zone associated with it. Use something like "internal.virtnet" (but not ".local") as your internal DNS zone.
I deviated here again. I have pfSense between my lab and my wireless so I setup my pfSense to handle the dhcp addresses for the lab. The dhcp service also passes out the ips for local dns servers, ntp server (handled by pfSense), and the tftp server. I installed a vm. Grabbed the post-install kickstart, tweaked it a million times, and threw it out to my ftp server. Now when I boot a vm that is pxe aware it will boot to a graphical menu (menu.c32 is for chumps), and allow me to select which OS to put on it.
4) Use that Spacewalk server to automatically (without touching it) install a new pair of OS instances, with which you will then create a Master/Master pair of LDAP servers. Make sure they register with the Spacewalk server. Do not allow anonymous bind, do not use unencrypted LDAP.
I've looked at openldap servers before, and there's a lot to them. You have Kerberos, SSSD, nsswitch, CAs, etc. I installed FreeIPA with DNS and made replicas. Oh, and my kickstart files will post install the ipa-client package so I can join them to the domain.
5) Reconfigure all 3 servers to use LDAP authentication.
Check
6) Create two new VMs, again unattendedly, which will then be Postgresql VMs. Use pgpool-II to set up master/master replication between them. Export the database from your Spacewalk server and import it into the new pgsql cluster. Reconfigure your Spacewalk instance to run off of that server.
Not on this yet, I have no experience with Postgresql. For the RHCE I'd probably go with MariaDB anyway.
7) Set up a Puppet Master. Plug it into the Spacewalk server for identifying the inventory it will need to work with. (**** and use ansible for deployment purposes, again plugging into the Spacewalk server.)
I've used Puppet in the past so I'm doing this with Saltstack at home.
8 ) Deploy another VM. Install iscsitgt and nfs-kernel-server on it. Export a LUN and an NFS share.
Using FreeNAS for a few years now to do this already to my VMware hosts. Also, targetcli is in 7 so I'd do that if I had to. I don't have NFS shares currently, but my todo would be NFS4 with Kerberos tickets and quotas.
9) Deploy another VM. Install bakula on it, using the postgresql cluster to store its database. Register each machine on it, storing to flatfile. Store the bakula VM's image on the iscsi LUN, and every other machine on the NFS share.
I'm tasked with doing Bacula for a client. It's feels cludgy, and there was some forks and lawsuits last year so not sure what the best way to go is. Backing up terabytes, I'd be inclined to rsync to offsite storage on ZFS or BTRFS (whenever they build in raid 6).
10) Deploy two more VMs. These will have httpd (Apache2) on them. Leave essentially default for now.
This is a RHCSA bullet item. I'm working on using Apache to do full SSO through FreeIPA / ldap.
11) Deploy two more VMs. These will have tomcat on them. Use JBoss Cache to replicate the session caches between them. Use the httpd servers as the frontends for this. The application you will run is JBoss Wiki.
Not here yet.
12) You guessed right, deploy another VM. This will do iptables-based NAT/round-robin loadbalancing between the two httpd servers.
I'd probably go HAproxy on this.
13) Deploy another VM. On this VM, install postfix. Set it up to use a gmail account to allow you to have it send emails, and receive messages only from your internal network.
Not inclined to touch email or printers. Because f-printers.
14) Deploy another VM. On this VM, set up a Nagios server. Have it use snmp to monitor the communication state of every relevant service involved above. This means doing a "is the right port open" check, and a "I got the right kind of response" check and "We still have filesystem space free" check.
This is what I'm tying into SSO with Apache. I'd like to have it automated. I've used this and Icinga in the past.
15) Deploy another VM. On this VM, set up a syslog daemon to listen to every other server's input. Reconfigure each other server to send their logging output to various files on the syslog server. (For extra credit, set up logstash or kibana or greylog to parse those logs.)
I customized a syslog server to do this a long time ago, but lost it in a power outage. It's on the shortlist to rebuild with an existing Splunk server.
16) Document every last step you did in getting to this point in your brand new Wiki.
Would like to do something with git instead and automate the documentation.
17) Now go back and create Puppet Manifests to ensure that every last one of these machines is authenticating to the LDAP servers, registered to the Spacewalk server, and backed up by the bakula server.
Do it with hiera for extra credit. I haven't seen many Puppet modules that are hiera aware. Foreman uses a different External Node Classifier (ENC) than hiera. I'd prefer to setup a git server and have all of your infrastructure changes there. Then, you could backup your repos, git, kickstarts and rebuild your whole infrastructure with Puppet, Salt, w/e.
18 ) Now go back, reference your documents, and set up a Puppet Razor profile that hooks into each of these things to allow you to recreate, from scratch, each individual server.
No idea what Razor is.
19) Destroy every secondary machine you've created and use the above profile to recreate them, joining them to the clusters as needed.
Todo
20) Bonus exercise: create three more VMs. A CentOS 5, 6, and 7 machine. On each of these machines, set them up to allow you to create custom RPMs and import them into the Spacewalk server instance. Ensure your Puppet configurations work for all three and produce like-for-like behaviors.
I'm working my way through CentOS 7 for this step. I don't know about going with 5, and would instead throw in an Ubuntu one.
The list doesn't really touch on bind. I'd recommend that you setup 2 dns servers with forward, reverse, and aliases. Understand what service principal names do.
FreeIPA is a wrapper for ldap, but also has some nifty functionality in place or coming down the line. It can host sshfp keys for your hosts so it doesn't ask "do you know this host...", automount keys, sudo rules, hbac, etc.
I didn't create the list though, I saw it on reddit. It shows up there as a "what should I learn for linux admins". Figured I'd move my buried reply to a new post. This is where I am on the list in italics, and how I would make some changes to it.
Note: Disable the firewall / SELinux to test your stuff works, but turn it on when you're done. You'll learn more that way. Ensure your services survive reboots. Take snapshots of your vms (unless they're Active Directory controllers...don't ask). I hate how guides online tell you to disable the security. You're lucky if all the flags they told you are correct also.
1) Set up a KVM hypervisor.
Yup
2) Inside of that KVM hypervisor, install a Spacewalk server. Use CentOS 6 as the distro for all work below. (For bonus points, set up errata importation on the CentOS channels, so you can properly see security update advisory information.)
I read into this and ended up installing a foreman server which appears to be replacing Spacewalk. It integrates Puppet and some of the functionality of Spacewalk to do provisioning and change management. It's similar to other cloud provisioning tools like OpenStack.
I'd recommend instead to build a vm and setup your own local mirror of the CentOS 6.6 repo, the epel repo, and configure a cron job to rysnc them weekly. I did that a few months ago, and wished I'd done it earlier.
3) Create a VM to provide named and dhcpd service to your entire environment. Set up the dhcp daemon to use the Spacewalk server as the pxeboot machine (thus allowing you to use Cobbler to do unattended OS installs). Make sure that every forward zone you create has a reverse zone associated with it. Use something like "internal.virtnet" (but not ".local") as your internal DNS zone.
I deviated here again. I have pfSense between my lab and my wireless so I setup my pfSense to handle the dhcp addresses for the lab. The dhcp service also passes out the ips for local dns servers, ntp server (handled by pfSense), and the tftp server. I installed a vm. Grabbed the post-install kickstart, tweaked it a million times, and threw it out to my ftp server. Now when I boot a vm that is pxe aware it will boot to a graphical menu (menu.c32 is for chumps), and allow me to select which OS to put on it.
4) Use that Spacewalk server to automatically (without touching it) install a new pair of OS instances, with which you will then create a Master/Master pair of LDAP servers. Make sure they register with the Spacewalk server. Do not allow anonymous bind, do not use unencrypted LDAP.
I've looked at openldap servers before, and there's a lot to them. You have Kerberos, SSSD, nsswitch, CAs, etc. I installed FreeIPA with DNS and made replicas. Oh, and my kickstart files will post install the ipa-client package so I can join them to the domain.
5) Reconfigure all 3 servers to use LDAP authentication.
Check
6) Create two new VMs, again unattendedly, which will then be Postgresql VMs. Use pgpool-II to set up master/master replication between them. Export the database from your Spacewalk server and import it into the new pgsql cluster. Reconfigure your Spacewalk instance to run off of that server.
Not on this yet, I have no experience with Postgresql. For the RHCE I'd probably go with MariaDB anyway.
7) Set up a Puppet Master. Plug it into the Spacewalk server for identifying the inventory it will need to work with. (**** and use ansible for deployment purposes, again plugging into the Spacewalk server.)
I've used Puppet in the past so I'm doing this with Saltstack at home.
8 ) Deploy another VM. Install iscsitgt and nfs-kernel-server on it. Export a LUN and an NFS share.
Using FreeNAS for a few years now to do this already to my VMware hosts. Also, targetcli is in 7 so I'd do that if I had to. I don't have NFS shares currently, but my todo would be NFS4 with Kerberos tickets and quotas.
9) Deploy another VM. Install bakula on it, using the postgresql cluster to store its database. Register each machine on it, storing to flatfile. Store the bakula VM's image on the iscsi LUN, and every other machine on the NFS share.
I'm tasked with doing Bacula for a client. It's feels cludgy, and there was some forks and lawsuits last year so not sure what the best way to go is. Backing up terabytes, I'd be inclined to rsync to offsite storage on ZFS or BTRFS (whenever they build in raid 6).
10) Deploy two more VMs. These will have httpd (Apache2) on them. Leave essentially default for now.
This is a RHCSA bullet item. I'm working on using Apache to do full SSO through FreeIPA / ldap.
11) Deploy two more VMs. These will have tomcat on them. Use JBoss Cache to replicate the session caches between them. Use the httpd servers as the frontends for this. The application you will run is JBoss Wiki.
Not here yet.
12) You guessed right, deploy another VM. This will do iptables-based NAT/round-robin loadbalancing between the two httpd servers.
I'd probably go HAproxy on this.
13) Deploy another VM. On this VM, install postfix. Set it up to use a gmail account to allow you to have it send emails, and receive messages only from your internal network.
Not inclined to touch email or printers. Because f-printers.
14) Deploy another VM. On this VM, set up a Nagios server. Have it use snmp to monitor the communication state of every relevant service involved above. This means doing a "is the right port open" check, and a "I got the right kind of response" check and "We still have filesystem space free" check.
This is what I'm tying into SSO with Apache. I'd like to have it automated. I've used this and Icinga in the past.
15) Deploy another VM. On this VM, set up a syslog daemon to listen to every other server's input. Reconfigure each other server to send their logging output to various files on the syslog server. (For extra credit, set up logstash or kibana or greylog to parse those logs.)
I customized a syslog server to do this a long time ago, but lost it in a power outage. It's on the shortlist to rebuild with an existing Splunk server.
16) Document every last step you did in getting to this point in your brand new Wiki.
Would like to do something with git instead and automate the documentation.
17) Now go back and create Puppet Manifests to ensure that every last one of these machines is authenticating to the LDAP servers, registered to the Spacewalk server, and backed up by the bakula server.
Do it with hiera for extra credit. I haven't seen many Puppet modules that are hiera aware. Foreman uses a different External Node Classifier (ENC) than hiera. I'd prefer to setup a git server and have all of your infrastructure changes there. Then, you could backup your repos, git, kickstarts and rebuild your whole infrastructure with Puppet, Salt, w/e.
18 ) Now go back, reference your documents, and set up a Puppet Razor profile that hooks into each of these things to allow you to recreate, from scratch, each individual server.
No idea what Razor is.
19) Destroy every secondary machine you've created and use the above profile to recreate them, joining them to the clusters as needed.
Todo
20) Bonus exercise: create three more VMs. A CentOS 5, 6, and 7 machine. On each of these machines, set them up to allow you to create custom RPMs and import them into the Spacewalk server instance. Ensure your Puppet configurations work for all three and produce like-for-like behaviors.
I'm working my way through CentOS 7 for this step. I don't know about going with 5, and would instead throw in an Ubuntu one.
The list doesn't really touch on bind. I'd recommend that you setup 2 dns servers with forward, reverse, and aliases. Understand what service principal names do.
FreeIPA is a wrapper for ldap, but also has some nifty functionality in place or coming down the line. It can host sshfp keys for your hosts so it doesn't ask "do you know this host...", automount keys, sudo rules, hbac, etc.
Comments
-
ChickenNuggetz
Member Posts: 284
Great list, notes, and edits! I think I saw the same post (comment) on Reddit a couple of months ago and was intrigued. I work heavily on RHEL systems so I've been thinking about picking up RHCSA/RHCE over the next year or so. I have an unused box at home that I could use as a sandbox. Let us know how this all pans out! I may end up picking your brain on some of these...:study: Currently Reading: Red Hat Certified Systems Administrator and Engineer by Ashgar Ghori
Certifications: CCENT; CCNA: R&S; Security+
Next up: RHCSA -
kly630
Member Posts: 72 ■■□□□□□□□□
Thanks for the post again Xavor. I think I found it once when you linked it in that thread, really liked the info/insight and decided to do it this year and track it in a blog. I probably mentioned it to brownwrap in another thread. It was really nice to have a roadmap laid out by someone that looked somewhat reasonable so that's a terrific find.
I'll be using ESXi 5.5 free to do my tasks. Hopefully I'm not too far off base picking that as my hypervisor but I'll figure out quickly.
As far as steps go, I have spacewalk installed fairly quickly on a VM as of yesterday. Probably need to make a post about the process shortly. Just as soon as I figure out whatever the errata importation is.
Edit:
Kind of wonder if it makes sense to post in this thread as we work through some of this? Maybe it'll inspire some other users to congregate and study redhat with us?
I'm pretty meticulous so I plan on trying everything, including the bullet points. I'm hoping if people read and ask questions my notes will get really good and I can update my blog posts to reflect them. -
Xavor
Member Posts: 161
ChickenNuggetz wrote: »Great list, notes, and edits! I think I saw the same post (comment) on Reddit a couple of months ago and was intrigued. I work heavily on RHEL systems so I've been thinking about picking up RHCSA/RHCE over the next year or so. I have an unused box at home that I could use as a sandbox. Let us know how this all pans out! I may end up picking your brain on some of these...
Yup, the original post was on reddit. My reaction was the same when I read it because starting out on a Linux admin path there are so many ways to go about doing something.
@kly630: Taking notes will really help retain the knowledge. The free ESXi server is fine for learning and is very straightforward to get going (caveat drivers, hardware compatibility).
Feel free to add whatever notes you want to this or wherever. I don't try to maintain a list or blog because I just don't have the time to make anything consistent. I've had a public webpage up for a year and there's nothing on it.
I think the strongest thing about a list like the above is the concepts it's teaching you. Get the basics, automate, document & change management, monitor, backups. It's easy to sit down and learn a lot of different vendor products, but not have those things in place. Knowing when something broke, root cause, prevention, and eventually automating the process away are pretty important.
I've already drifted from that list myself since I posted the edits. I nuked my Foreman server and built a Salt-Master.
I built a second storage server in an attempt to replace my FreeNAS with linux. FreeNAS uses ZFS and I have it set up as a RAIDZ2 (RAID 6). I wanted to do something with btrfs, but I'm not there yet.
I did, however, build a CentOS 6 server with NFS 3, 4, iSCSI, software raid 6 on lvm, and S.M.A.R.T. monitoring tools. This mimics what I use in FreeNAS already, but I still need to setup MRTG graphing, email alerts to my real email, and UPS monitoring for alerts. Each of those steps was a learning process because FreeNAS does it pretty cleanly, but in linux you may spend a few hours on configuring each step if you've never done it.
I'd like to configure NFS 4 with kerberos, but again, time.
Oh, and I found out there's a free demo for the Cisco UCS as VM so...fml. -
kly630
Member Posts: 72 ■■□□□□□□□□
I finally wrote up a small post about installing spacewalk. It's so well automated I basically had to do very little.
Afterward, I set up a new software channel, added the centos 6 base repo, and created an activation key. In installed a few rhn-tools type packages and ran rhn_register and things registered.
Looks like a really nice way to sync/monitor packages and config files on the server. I'm also pretty excited to use some of the kickstart features. -
asummers
Member Posts: 157
I finally wrote up a small post about installing spacewalk. It's so well automated I basically had to do very little.
Afterward, I set up a new software channel, added the centos 6 base repo, and created an activation key. In installed a few rhn-tools type packages and ran rhn_register and things registered.
Looks like a really nice way to sync/monitor packages and config files on the server. I'm also pretty excited to use some of the kickstart features.
Not sure if it's changed - but when I installed I had to do some extra stuff not currently on your blog.
- NTP needs to enabled
- selinux perms on (/var/lib/pgsql and /var/satellite)
- Configure firewall
- Configre EPEL repo -
Xavor
Member Posts: 161
Glad this gave you some motivation. I'm still working on rebuilding my storage server in CentOS 7 from 6.
-
Kinet1c
Member Posts: 604 ■■■■□□□□□□
Glad this gave you some motivation. I'm still working on rebuilding my storage server in CentOS 7 from 6.
I've got that very post saved for the summer time, hopefully get through all of it and then start working towards my RHCSA later this year.2018 Goals - Learn all the Hashicorp products
Luck is what happens when preparation meets opportunity -
kly630
Member Posts: 72 ■■□□□□□□□□
You're right. I just set all my servers to use ntp when I first install. And I'm being really lazy and turning off selinux and iptables to start, but in the interest of completeness I'll go back and modify the post and add what's needed to get it running with those things. Definitely want it to stand as the best set of notes I can put together.Not sure if it's changed - but when I installed I had to do some extra stuff not currently on your blog.
- NTP needs to enabled
- selinux perms on (/var/lib/pgsql and /var/satellite)
- Configure firewall
- Configre EPEL repo -
asummers
Member Posts: 157
I think the general rule is to never turn off selinux and iptables in the exam. You should practice with both on (and go through the pain that causes).
I'll be keeping an lookout for your blog
Are you building a spacewalk server for the EX401 exam or for practice? if not doing EX401 then I would suggest ditching spacewalk and using creating your own build server (http/dhcp/pxe etc)
