Road to RHCE

Verities

wolfinsheepsclothing wrote: »

^Keep it up! As for iSCSI (on the target server), I recommend using the targetcli interactive shell (takes a few standard commands like: cd,ls,pwd) to configure the target/acl/lun/portal. Quick example (assuming you're feeding out a block device and not a file):

targetcli

>cd backstores
backstores> block/ create nameOfYourChoosing /dev/yourBlockDevice
backstores> cd /iscsi
iscsi> create iqn.2016-05.com.example:serverName
iscsi> cd iqn.2016-05.com.example:serverName/tpg1 (<----you should be able to tab complete this)
iscsi/iqn.2016-05.com.example:serverName/tpg1> acls/ create iqn.2016-05.com.example:yourClient
iscsi/iqn.2016-05.com.example:serverName/tpg1> luns/ create /backstores/block/nameOfYourChoosing (<---same name as earlier; just tab complete it)
iscsi/iqn.2016-05.com.example:serverName/tpg1> portals/ create yourServerIP


With regard to the rich rules, if you don't recall specifics:

man firewalld.richlanguage (there are examples at the bottom)

Thanks for the info, but I'm sure I'll procrastinate until the very end for iSCSI. Fortunately, I've been implementing rich rules in production when deploying new RHEL 7 VMs and after doing a few, I've found its easy to accomplish what you want it to do. We're also using ipv6 in our environment so I actually get to implement ipv6 rich rules which is kind of cool.

Verities

I've been reviewing the objective for teaming or bonding the NICs and was curious about the different features of both. I came across an interesting RHEL Blog that covered both:

If You Like Bonding, You Will Love Teaming – Red Hat Enterprise Linux Blog

Verities

Today I'm reviewing: configuring a system to authenticate with Kerberos.

First step is setting up a KDC (using FreeIPA), then the actual authentication part is very easy with authconfig-tui. Authconfig is excellent in that it will let you know if you need to install any additional packages (usually pam_krb5). Also, if you don't have DNS configured this will not work properly.

varelg

Cert Depot guides helped me a lot when having to set up remote share servers and, among other things, kerberize them. That was totally out of the list of RHCSA objectives but I got myself a nice setup against which I could verify my client setup for remote shares.
The sad part is that it's sooooo easy to forget to run kinit user_name before starting the exercises...

Verities

varelg wrote: »

Cert Depot guides helped me a lot when having to set up remote share servers and, among other things, kerberize them. That was totally out of the list of RHCSA objectives but I got myself a nice setup against which I could verify my client setup for remote shares.
The sad part is that it's sooooo easy to forget to run kinit user_name before starting the exercises...

Yeah I used a few of his tips for both RHCSA and for some RHCE objectives. You may find it interesting that he (cert depot guy) actually was a technical reviewer for Sander Van Vugt's RHCSA/RHCE book. After reading that I noticed a lot of his material is similar to Sander's. In this case, I used Sander's steps from his video to setup free ipa server and then configuring authentication with Kerberos.

Good point on the kinit command as you want to make sure its actually working.

varelg

Is it a normal/default behavior for CentOS/RHEL 7 clients to download a generic krb5.conf file along with the keytab file when they deal with a Kerberized server? Or is it maybe the way servers are set up at Cert Depot's tutorials, like an outdated NFS or Kerber version? I had to deliberately download server's version of krb5.conf file along with the keytab or client would not authenticate.

Verities

varelg wrote: »

Is it a normal/default behavior for CentOS/RHEL 7 clients to download a generic krb5.conf file along with the keytab file when they deal with a Kerberized server? Or is it maybe the way servers are set up at Cert Depot's tutorials, like an outdated NFS or Kerber version? I had to deliberately download server's version of krb5.conf file along with the keytab or client would not authenticate.

You were correct in what you did; when it comes to applications like NFS, you have to manually copy the keytab from the NFS server to the clients, if using krb5 for authentication or else they can't authenticate. I just looked at his steps on the cert depot page and indeed he is missing that step.

varelg

No, not keytab, it was /etc/krb5.conf that was confusing- as a part of Kerberizing the server you'd edit /etc/krb5.conf file to reflect which domain is Kerberos in charge of. But when you configure the client to log in to the kerberized server and you download the keytab, you are also getting a krb5.conf file that was not edited and is generic, with EXAMPLE.COM as a domain. How on earth did the generic krb5.conf file got in there, instead of the krb5.conf that was adjusted for the actual domain?! Would it be CentOS vs. RHEL quirk, or NFS versioning maybe?

Verities

I used Sander's configuration for configuring a KDC, which uses Free-Ipa. I don't have experience configuring a KDC with KRB5-server, which is what Cert Depot suggests (it looks like a lot more steps). Make sure your DNS is configured so that you can communicate and resolve the hostname of your KDC.

Having said that, all you need to authenticate to a system using Kerberos once you have a Free-Ipa KDC setup, is install pam_krb5 and authconfig (tui,gtk,w/e) on the client. Then use authconfig-tui since its the fastest way to configure the settings. Authconfig will do the configuration of your /etc/krb5.conf file and if you entered in the right information will have your server point to your respective KDC. This process satisfies the RHCE objective "configure a system to authenticate using Kerberos".

The other RHCE objective you may be talking about is for NFS "use kerberos to control access to network shares" should be built upon how you configured your original KDC. After reviewing the Cert Depot steps, its very close to what Sander suggests for configuring NFSv4 and having the clients authenticate to the share. However, Sander's setup uses free-ipa client, which again is different if you're using KRB5-server so I'm not sure how that would affect your NFS setup. I used NFSv4 and only had to copy the /etc/krb5.keytab file to the clients trying to access the share.

I highly suggest going with Sander's method. I will test Cert Depot's method either later today or tomorrow morning (depending on when I have time to use my lab) and get back to you if its your setup that's misconfigured or Cert Depots.

crylium

I configured Kerberised NFS on all RHEL versions (7.0, 7.1 and 7.2), and the keytab file was the only one that needed copying.

Verities

Next week I'll be back to studying for the exam, covering everything I need to brush up on. I'm looking to sit for the RHCE exam around August 10th.

Verities

So I finally got around to comparing the krb5-server vs Free-IPA server. I highly recommend using Free IPA during the exam because you receive a working KDC right out of the box. Using the krb5-server package by itself actually requires a significant amount of work to setup. I believe going through all the motions to setup a KDC is worth it for learning purposes, but due to the time constraints on the exam I will use Free IPA.

Verities

When labbing (Linux Academy) out the following task I ran into something unusual:

Configure a system as either an iSCSI target or initiator that persistently mounts an iSCSI target

I configured the target and initiator, then when I went to discover the portal for the target everything was fine. I then noticed once I installed firewalld on the iSCSI target server and add 3620/tcp to the firewall, I was no longer able to find the portal from my initiator:


iscsiadm: cannot make connection to 10.0.0.100: No route to host

So with firewalld runnig and the port listening (test using ss-arn | grep 3620), I figured there is something missing from the firewall configuration. I looked at the known services of firewalld using:


firewall-cmd --get-services

and what do you know there's an "iscsi-target"....so I went to take a look at the XML file that defines the service for firewalld:


cat /usr/lib/firewalld/services/iscsi-target.xml
<?xml version="1.0" encoding="utf-8"?>
<service>
<short>iSCSI target</short>
<description>Internet SCSI target is a storage resource located on an iSCSI server.</description>
<port protocol="tcp" port="3260"/>
<port protocol="udp" port="3260"/>
</service>

Voila! UDP needs to be added to the port number, which isn't in any of the material from Linux Academy or Sander's Videos/book. After adding the known service it worked fine, but its strange to me that it worked in all of these videos I've watched when they only added 3620/tcp yet I needed to add the udp protocol for that port. Moving on to Network services/SELinux for a refresher.

SELinux shortcut(throw it in a script and make it executable):


cat /var/log/audit/audit.log | audit2allow -m local > local.te; cat local.te

hiddenknight821

Thanks for sharing. It's pretty cool how you came across this. Even though I haven't taken the RHCSA, I'm pretty much ready to take it as I'm in the reviewing phase for the next few weeks. I'm already anxious to get started on the RHCE topic.

What drew me to your post was this.

Verities wrote: »

SELinux shortcut(throw it in a script and make it executable):


cat /var/log/audit/audit.log | audit2allow -m local > local.te; cat local.te

Seeing this, I knew it's a quick and dirty workaround as confirmed in the warning here. So we are not expected to write an SELinux policy for RHCE exam, right?

Verities

No problem. I figured the best thing to do while studying is communicate the bumps I run into and hope others can learn along the way as well. Training material and official documentation are fine and dandy, but labbing is where the rubber meets the road.

I use the script to identify what policies are necessary, if at all. I've used it for a few policy violations that helped me quickly identify what booleans needed to be enabled. I have yet to have it pose a significant threat on my systems. The real threat comes when you turn off SELinux (DON'T EVER TURN IT OFF...unless you're using HIPS in place of it) because you can't figure out a policy violation.

Here are the objectives:

Configure SELinux to support the service


Use SELinux port labeling to allow services to use non-standard ports

The great thing about Red Hat exams is they only care about the end product: did you meet the objective? If yes, then points If no, then no points

varelg

At the risk of earning a negative reputation, I'd go for the low-laying fruit and ask: did you run firewall-cmd --reload after you added the needed port and was that zone where port was added the active zone?

hiddenknight821

Verities wrote: »

No problem. I figured the best thing to do while studying is communicate the bumps I run into and hope others can learn along the way as well. Training material and official documentation are fine and dandy, but labbing is where the rubber meets the road.

Very true. The thing I love about the exam is that there is more than one way to solve the problem. Shortly after you made your post, I noticed CertDepot tweeted about the same problem you encountered.

varelg wrote: »

At the risk of earning a negative reputation....

Neg-rep members here is rather silly IMO as long as they're not being too obnoxious on the forum. You asked a good question. I'm sure we all make mistakes and overlooked some minor details. No matter how experienced or knowledgeable we are. I like TE because we genuinely help each other out.

Verities

varelg wrote: »

At the risk of earning a negative reputation, I'd go for the low-laying fruit and ask: did you run firewall-cmd --reload after you added the needed port and was that zone where port was added the active zone?

The known service file (library) that was created by Red Hat for iSCSI-target had both ports being required. You had a valid question and to answer that question, yes I did do a reload. iSCSI-target needs to communicate over UDP and TCP for port 3620. There is no mention of the port in LIO-Target which is wierd so I believe the UDP portion it has something to do with how the target is discovered (think of DORA for DHCP). To know for sure would be to run a tcpdump on the port and see what traffic is sent over UDP. I'm not that interested in finding out as I'm OK with my current theory.

vwtech

I created a KDC (example from CertDepot) and was able to setup krb5p share and mount it on a client.
Also setup an IPA server so I can use it for LDAP if needed.

Issues I encountered:
Problem is when I configure kerberos authentication to my IPA server. Configured host and nfs entries in keytab. I could authenticate users but when I attempted to mount nfs with krb5p I got the following error:

[mount.nfs: access denied by server while mounting server1:/nfs-secure]

I don't know if we'll be able to use ipa-client to configure authentication since admin credentials to the IPA server are required to do so.
For now-I'm going to use the KDC server for kerberos since that actually works for me.
If anyone has help on this issue - it's welcomed.

Verities

vwtech wrote: »

I created a KDC (example from CertDepot) and was able to setup krb5p share and mount it on a client.
Also setup an IPA server so I can use it for LDAP if needed.

Issues I encountered:
Problem is when I configure kerberos authentication to my IPA server. Configured host and nfs entries in keytab. I could authenticate users but when I attempted to mount nfs with krb5p I got the following error:

[mount.nfs: access denied by server while mounting server1:/nfs-secure]

I don't know if we'll be able to use ipa-client to configure authentication since admin credentials to the IPA server are required to do so.
For now-I'm going to use the KDC server for kerberos since that actually works for me.
If anyone has help on this issue - it's welcomed.

This sounds like a permissions issue but I still have to ask..

Did you create the service principal on your KDC for the server that has nfs? After that you generate a key tab and copy it to your nfs host and client.

If you've already made sure NFS and Kerberos have been added to the firewall and ensured there are no SELinux file context errors, then verify your /etc/export settings and when you're attempting to mount it from your client, ensure you're specifying the proper NFS version and protocol with which to mount it.

/etc/exports should look like this:

/nfs-secure *(sec=krb5p,rw)

export the file system (exportfs -avr), then test with a showmount -e

on the client you should be mounting with this:

mount –o sec=krb5p hostname:/nfs-secure /mnt/nfs-secure

brombulec

(a little off topic)
I'm looking on this thread and I wonder why are you so angry on Sander or other author for omitting the port number in books?
You want to be an RHCE (E = engineer) and engineer should know that if he configures the NETWORK serwice (such as iSCSI or HTTPS) he should OPEN specific ports on firewall. The exam blueprint says that you can't disable firewall and you have to run network services. Ergo, you have to "drill the hole" in the firewall and therefore you HAVE TO know which port and protocol is needed.
The engineer should be curious, should look for the answers not in one and only book - the Internet is so rich, so powerful and even the defeinition of iSCSI on Wikipedia shows that the iSCSI is using the 3260 port.

I assume that you want to go for the other RH certs (such as 413, 436) - there is no good book for these courses and the preparation for the RHCE should be a good testing ground for all engineers - you can test your google skills, look at the logs and not base only on the book. The book is not available on the exam, only you, your knowledge, VMs and MAN pages

And one more thing: you can always use

firewall-cmd --add-service=iscsi-target --permanent

And you don't have to remember those "ugly" ports and protocols

vwtech

Verities wrote: »

This sounds like a permissions issue but I still have to ask..

Did you create the service principal on your KDC for the server that has nfs? After that you generate a key tab and copy it to your nfs host and client.

If you've already made sure NFS and Kerberos have been added to the firewall and ensured there are no SELinux file context errors, then verify your /etc/export settings and when you're attempting to mount it from your client, ensure you're specifying the proper NFS version and protocol with which to mount it.

/etc/exports should look like this:

/nfs-secure *(sec=krb5p,rw)

export the file system (exportfs -avr), then test with a showmount -e

on the client you should be mounting with this:

mount –o sec=krb5p hostname:/nfs-secure /mnt/nfs-secure

The KDC server (exports the nfs) and the IPA server that exports the nfs share are two different nodes.
I reviewed the keytab for IPA server and nfs/kdc.example.com@EXAMPLE.COM is present
The keytab for KDC server and nfs/kdc.example.com@EXAMPLE.COM also present
SElinux context on the directory and files is "public_content_t"

To your point; I ensured there was a nfs service principle for each node in the keytab of the ipa server, configured nfs again and got the same error.
Question: Did you use the join the ipa domain with the "ipa-client-install" command ?

Verities

brombulec wrote: »

(a little off topic)
I'm looking on this thread and I wonder why are you so angry on Sander or other author for omitting the port number in books?
You want to be an RHCE (E = engineer) and engineer should know that if he configures the NETWORK serwice (such as iSCSI or HTTPS) he should OPEN specific ports on firewall. The exam blueprint says that you can't disable firewall and you have to run network services. Ergo, you have to "drill the hole" in the firewall and therefore you HAVE TO know which port and protocol is needed.
The engineer should be curious, should look for the answers not in one and only book - the Internet is so rich, so powerful and even the defeinition of iSCSI on Wikipedia shows that the iSCSI is using the 3260 port.

I assume that you want to go for the other RH certs (such as 413, 436) - there is no good book for these courses and the preparation for the RHCE should be a good testing ground for all engineers - you can test your google skills, look at the logs and not base only on the book. The book is not available on the exam, only you, your knowledge, VMs and MAN pages

And one more thing: you can always use

firewall-cmd --add-service=iscsi-target --permanent

And you don't have to remember those "ugly" ports and protocols

Based on your post I've gathered you either didn't read everything I typed out and jumped straight to a completely inaccurate conclusion or English isn't your first language so you didn't understand. I hope its the latter since your location says Poland.

I never said I was angry at either of the content providers, I simply stated they did not include certain information in their training materials. As for the issue with the iscs-target..I identified the problem, figured out the cause, and the solution, all on my own. I thought I did a pretty good job explaining my thought process and how I came to a resolution.

Also, thanks for reiterating what is and what is not available on the Red Hat exams, however I am familiar with the format and objectives.

Verities

vwtech wrote: »

The KDC server (exports the nfs) and the IPA server that exports the nfs share are two different nodes.
I reviewed the keytab for IPA server and nfs/kdc.example.com@EXAMPLE.COM is present
The keytab for KDC server and nfs/kdc.example.com@EXAMPLE.COM also present
SElinux context on the directory and files is "public_content_t"

To your point; I ensured there was a nfs service principle for each node in the keytab of the ipa server, configured nfs again and got the same error.
Question: Did you use the join the ipa domain with the "ipa-client-install" command ?

Yes I understand you're using two different nodes but the initial configuration is the same in regards to principals and keytabs. On the client I did use the ipa-client-install (along with) -enable-dns-updates.

vwtech

Verities wrote: »

Yes I understand you're using two different nodes but the initial configuration is the same in regards to principals and keytabs. On the client I did use the ipa-client-install (along with) -enable-dns-updates.

The major question is:
Will they provide us with credentials in order to join a client the a ipa domain ?
If the exams doesn't; we can just use authconfig-tui or authconfig to enable krb5 auth, connect to the KDC, install krb5-workstation, get the krb5.keytab.

Verities

vwtech wrote: »

The major question is:
Will they provide us with credentials in order to join a client the a ipa domain ?
If the exams doesn't; we can just use authconfig-tui or authconfig to enable krb5 auth, connect to the KDC, install krb5-workstation, get the krb5.keytab.

Excellent question, I would imagine they do provide you with credentials for authconfig since there is no objective to setup a KDC for the exam. According to the objectives you need only to be able to configure a server to authenticate Kerberos and use Kerberos to control access to NFS network shares.

varelg

@vwtech: so after you kerberized your server you lost the ability to mount shares, am I reading it correctly? you may want to compare krb5.conf files on both client and server, are they the same? They should be, for successful mount. Also, run kinit on the server before attempting anything with clients. i encountered similar quirk while setting up NFS server from Cert depot, some Kerberos server default settings may differ between CentOS and Red Hat.

Verities

I've been off my RHCE studies for a bit, but finally finished the RHCE course on Linux Academy, so I'm happy about that. I really have no desire to get my RHCE, but I know I'll get a salary bump and my company is going to pay for the exam if I pass. I've been focusing on Ansible more than anything so I can use a central point of control for my environment and keeping configurations as standardized as possible. Its pretty awesome to manage everything from 1 server and I've even managed to write a few solid playbooks: updating iptables and firewalld with new ipv4/ipv6 rules then removing old ones as well as automating other security tasks (server hardening).

Taking the actual exam is low on my priority list, but I will take it eventually. I've suddenly become responsible for a number of technologies that I don't have in-depth experience with that I now have to maintain and troubleshoot in a timely manner. Also, my workload has increased since the only other person I work with was recently moved to another project, so I've got double the work with no help. It was scary at first, but I've come to enjoy the challenge and I'm finding my technical abilities are progressing to higher levels much quicker than before.

asummers

Verities wrote: »

I've been off my RHCE studies for a bit, but finally finished the RHCE course on Linux Academy, so I'm happy about that. I really have no desire to get my RHCE, but I know I'll get a salary bump and my company is going to pay for the exam if I pass. I've been focusing on Ansible more than anything so I can use a central point of control for my environment and keeping configurations as standardized as possible. Its pretty awesome to manage everything from 1 server and I've even managed to write a few solid playbooks: updating iptables and firewalld with new ipv4/ipv6 rules then removing old ones as well as automating other security tasks (server hardening).

Taking the actual exam is low on my priority list, but I will take it eventually. I've suddenly become responsible for a number of technologies that I don't have in-depth experience with that I now have to maintain and troubleshoot in a timely manner. Also, my workload has increased since the only other person I work with was recently moved to another project, so I've got double the work with no help. It was scary at first, but I've come to enjoy the challenge and I'm finding my technical abilities are progressing to higher levels much quicker than before.

Only thing I would say is that ultimately you can only hold so much in your mind at any one time and if you have trained for an exam then do the exam. The hard part is the studying which you have done - seems a waste not to put it to use.

hiddenknight821

^^^^^ Agreed. Ever played Mortal Kombat? FINISH HIM!

Just do the cert and get this over with. We've seen how much work you spent on this cert, and we'd hate to see your hard work go to waste.