RHCE Objectives: Sed and Awk

When you start working in environments with hundreds of Linux and Unix machines sed & awk can be used take away some of the manual CLI hashing time. Sed and awk can also be used when you start automating your environment. Using scripts to pull data out of text files then piping or passing the results to a script or program which automates your work for you.

An example would be when you create scripts you often do not want to input every IP, hostname, kernel version and so on instead you have the script crawling across your server farm find it out for you.

As to answer your question about the RHCE:
The use of sed & awk are not mandatory for the exam but some type text parsing or text redirect knowledge is. You have 3.5 hours to complete the practical which involves configuration. They create a situation and you have to fix problems and configure features. I have not taken the test (not offered in Japan) but I have been to the class.

The sticky in this forum for the RHCE study material is pretty good.

lumbercis

Thank you very much!
In looking at the exam objectives again, they have sed and awk listed under "prerequisite skills." They don't say so directly, but this could be taken to mean that these areas are not a direct focus of the test. I think I'll just make sure to know the basic sed and awk operations/commands just in case they pop up on the test.

Thanks again!
J

You should be pretty good with that. Remember the biggest killer with this exam is time so you need to be fast and accurate. I work with a bunch of RHCE and the ones who did not make it during their test or those who did not make it the first time said they injected problems which caused cascading side effects. They then spent too much time on those problems that they created.

I'm facing the same problem as I'm preparing for RHCE...I don't use sed & awk very often to be honest since my job doesn't demand scripting a lot.

I skipped this topic, and now going through the last chapters.

Personally I'll go with the depth mentioned in the Michael Jang book, it seems to be focusing on the exam itself.

Good website with lots of resources:
http://www.tldp.org/LDP/abs/html/x16155.html

Some Sed examples:
Part 1 Common threads: Sed by example, Part 1
Part 2 Common threads: Sed by example, Part 2
Part 3 Common threads: Sed by example, Part 3

Some Awk examples:
Awk

A Sed Tutorial:
Sed - An Introduction and Tutorial

An Awk Tutorial:
Awk Tutorial and Introduction

lumbercis

^Nice.

I especially liked the tutorials on the ibm.com site for being very concise, yet thorough.

Has anyone found a good site with sed&awk exercises? I find that things gel best for me when I'm put on the spot with exercises where I'm forced to think about how to apply what I've just been shown.

Thanks
J.

EDIT: http://discuss.joelonsoftware.com/default.asp?joel.3.577929.18 I found this discussion interesting as an example of the variety of ways people are using (or not using) sed and awk in the field. For some reason that makes me feel better. I'm such a global learner it drives me nuts!

varelg

Unix.com has an active (or at least they were active) forum with many calls for help with sed and awk scripts.

lumbercis wrote: »

Hi all,

I have an entire book dedicated to it (O'Reilly's "Sed&Awk") but I don't want to spend too much time becoming a sed and awk guru if the exam will only expect basic operations.

Please understand that what I'm about to say isn't meant to be as harsh as it sounds.

This is a lousy attitude. You should never be studying to pass an exam, you should be studying to acquire the knowledge. If you only learn enough to pass the exam, and then get into a job where you can't function if more advanced is required, you devalue the certification. This goes for all exams.

In particular to working in the unix world, sed and awk will be some of the best friends you have once you start administering live production servers. If you intend to make a career of this, learn them in and out and it will pay off for you down the road.

Let me give you some practical, albeit simple examples.

awk is extremely useful for formatting log output. In particular, it's always been a favorite of mine to pull open a window, tail an apache log file, awking to display only the result code and the referrer. Then, as I start making access changes, I can immediately see if the expected result code is popping up (ie, if I'm blocking hotlinkers, I can immediately tell if the rule is effective, because the result codes go from 200 to 403)

Last week, it was discovered that some fool had added a configuration directive to the wrong file. On about 300 servers. So I whipped up a little script to grep that directive out of the original file, append it to the correct file, then sed the directive out of the original file. Add in a for loop to login to each server and run that script, and it took about 10 minutes to correct. Doing it all by hand would have been a full day thing.

And that's really just child's play. sed and awk can basically take any text file and manipulate it in any way you wish. you just have to learn enough to make it so. The side education you get in regex as a course of learning sed and awk is also nothing to sneeze at.

I agree with you to some extent.

The world of unix - as you already know - is HUGE, and no one can really know everything. It really depends on your exposure and the nature of you work you do.

Of course, it's good to study and test things on your own lab, but nothing beats the real deal.

RHCE topics are tricky in the sense that the depth is not clear. For example, the topic of Apache can span multiple books, and the topic of sendmail is huge too. Unless you work as a sendmail admin, you can never be experienced in that.

Not all Unix related jobs would require in-depth text manipulation. For me, I usually deal with SAN and clusters and backups, and rarely do I face a problem where a script is even needed. But that doesn't mean that this knowledge isn't important and won't come in handy some day.

It's on my plan to sharpen my scripting (incuding sed/awk) knwoeldge, specially that I have very strong programming background (C,C++, assembly, C#.NET,ASP.NET,little bit of Java)...but for the moment, I'm gonna pass the RHCE. I already went deep with some topics such as LVM, RAID...but I don't think I'll go very deep with sed & awk right now.

Forsaken_GA wrote: »

Please understand that what I'm about to say isn't meant to be as harsh as it sounds.

This is a lousy attitude. You should never be studying to pass an exam, you should be studying to acquire the knowledge. If you only learn enough to pass the exam, and then get into a job where you can't function if more advanced is required, you devalue the certification. This goes for all exams.

In particular to working in the unix world, sed and awk will be some of the best friends you have once you start administering live production servers. If you intend to make a career of this, learn them in and out and it will pay off for you down the road.

Let me give you some practical, albeit simple examples.

awk is extremely useful for formatting log output. In particular, it's always been a favorite of mine to pull open a window, tail an apache log file, awking to display only the result code and the referrer. Then, as I start making access changes, I can immediately see if the expected result code is popping up (ie, if I'm blocking hotlinkers, I can immediately tell if the rule is effective, because the result codes go from 200 to 403)

Last week, it was discovered that some fool had added a configuration directive to the wrong file. On about 300 servers. So I whipped up a little script to grep that directive out of the original file, append it to the correct file, then sed the directive out of the original file. Add in a for loop to login to each server and run that script, and it took about 10 minutes to correct. Doing it all by hand would have been a full day thing.

And that's really just child's play. sed and awk can basically take any text file and manipulate it in any way you wish. you just have to learn enough to make it so. The side education you get in regex as a course of learning sed and awk is also nothing to sneeze at.

UnixGuy wrote: »

I already went deep with some topics such as LVM, RAID...but I don't think I'll go very deep with sed & awk right now.

Oh, the irony of someone with the handle of Unixguy downplaying the usefulness of sed and awk hehe

I just wish that folks could understand that by dumbing themselves down to the level an exam is asking for, they're cheating themselves. I'll just never understand the mentality of 'good enough'

Forsaken_GA wrote: »

Oh, the irony of someone with the handle of Unixguy downplaying the usefulness of sed and awk hehe...

wow I was downplaying the usefulness of sed & awk

I said I didn't go deep with the topics of sed & awk AND I don't use them much at my current job...that doesn't mean at all that such topics are not "important". I don't find it a big deal to be honest...I wasn't interested enough and so I skipped it.

If there's a need in my job to use sed & awk, then trust me I'm gonna use them, I did enough programming on C/C++/C#; string manipulation is the least challenging/interesting topic to me...but I might forget the syntax if I don't use it very often, that was my point. Just as someone might forget the exact options with commands such as installboot bootblk on Solaris for example...but you can find them if you want, as long as you know the concept and know when to use it.

As I said earlier, "Unix(and IT in general) world" is VERY broad..just because someone's job demands a certain amount of string manipulation doesn't mean that ALL Unix/IT-related jobs need this skill (that's obviously not an attempt to undermine the "string manipulation skill".

I don't think RHCE objectives are clear enough either, because as the OP said, you can find huge books talking about sed,awk, regex, ...etc...and he is right. It's natural that one would be strong in some areas and weak in some other areas. At the end of the day, it's just a test and a learning experience.

@UnixGuy I would not say the RHCE objectives are vague just not specific for the test. They do that because they want people at a certain aptitude to take the test and pass. This also encourages people to take the expensive class.

NightShade03

Chris:/* wrote: »

@UnixGuy I would not say the RHCE objectives are vague just not specific for the test. They do that because they want people at a certain aptitude to take the test and pass. This also encourages people to take the expensive class.

It also leaves the exam open to a more free form approach. For example they prob won't tell you to implement Sendmail because it isn't an objective, but they might ask you to implement something that provides SMTP functionality. Makes it easier for those that have experience with different technologies if you ask me.

UnixGuy wrote: »

I don't think RHCE objectives are clear enough either, because as the OP said, you can find huge books talking about sed,awk, regex, ...etc...and he is right. It's natural that one would be strong in some areas and weak in some other areas. At the end of the day, it's just a test and a learning experience.

I'm sorry dude, I can't quite buy that when it comes to unixland. You are, of course, entitled to your opinion, but in my opinion, sed,awk,grep,find, regex voodoo, etc etc etc are all foundational skills, not areas of specialization. I would never consider letting someone without those basics get anywhere near my production boxes

NightShade03 wrote: »

It also leaves the exam open to a more free form approach. For example they prob won't tell you to implement Sendmail because it isn't an objective, but they might ask you to implement something that provides SMTP functionality. Makes it easier for those that have experience with different technologies if you ask me.

That's true! I tend to fix sendmail and exim problems by installing postfix!

@NightShade03 Absolutely and I believe it is why the certification holds so much respect because of the level of knowledge and adaptability the candidate must have.

NightShade03

Chris:/* wrote: »

@NightShade03 Absolutely and I believe it is why the certification holds so much respect because of the level of knowledge and adaptability the candidate must have.

Now if only we could get Microsoft to convert

It will never happen ;P

Forsaken_GA wrote: »

I would never consider letting someone without those basics get anywhere near my production boxes

interesting, I think a lot of people share the same opinion with you. Maybe I shouldn't share my opinion with future potential employers

varelg

Chris:/* wrote: »

@UnixGuy I would not say the RHCE objectives are vague just not specific for the test. They do that because they want people at a certain aptitude to take the test and pass. This also encourages people to take the expensive class.

The class $$$ is more likely motivation behind "vagueness" than the candidates' aptitude. It would be interesting to see what's the share of cert related revenues in the overall Red Hat's revenue. I don't think it's something Red Hat would easily give up on.
You wouldn't let people near production boxes without certain level of knowledge Forsaken, but people with worse levels or no sed awk knowledge (or scripting solutions in general) ARE sysadmins, it's not something that you won't see. Like it or not, that IS happening.
Reading these few responses above, one would ask about the whole point of getting tested for this exam. I hope you won't equate it to the "who's peeing furthest" game.
I'd consider this certification only if it's employer- mandated and paid for. Learning experience? Yeah, but so is just reading the RH book given at the RH classes. In that retrospect Microsoft is more open than Red Hat: anyone can buy MS exam guides. Why should RH guide be any different, because it holds the secret spell?

NightShade03

I think there is something to be said about the Red Hat classes though. I hear that the in person one is pretty decent, but as for the online one (which I have taken), I can honestly say it is almost a scam. The online course was so horrible I can't believe that I wasted money on it.

I don't think the vaugeness of the prep guide is to encourage the course taking because bad reviews of any course will deter people away....but I am curious too how much revenue Red Hat takes in from those courses.

lumbercis

@Forsaken,

I appreciate your insight and agree with you for the most part. But I would bet that if you look back on how YOU became an expert at sed and awk (and sysadmin in general) you will not have (before you even had a job in IT) picked a singular task, studied it exclusively until you had complete command, then moved on the next discrete skill, studied it exclusively until you were an expert....etc. No one becomes an expert like that.

They gather basic skills, then gain most of their actual expertise from using that skill in an appropriate production context to solve real problems.

That's all I am suggesting. I certainly never meant to imply that I or anyone should skip sed and awk skills entirely since it wasn't on the test.

J.

MentholMoose

varelg wrote: »

I'd consider this certification only if it's employer- mandated and paid for. Learning experience? Yeah, but so is just reading the RH book given at the RH classes. In that retrospect Microsoft is more open than Red Hat: anyone can buy MS exam guides. Why should RH guide be any different, because it holds the secret spell?

Actually you can't buy the material used in Microsoft classes. You have to take the class. This is common for training courses, not just Red Hat and Microsoft. It just happens that Microsoft also publishes study guides, whereas Red Hat doesn't.

lumbercis wrote: »

@Forsaken,

I appreciate your insight and agree with you for the most part. But I would bet that if you look back on how YOU became an expert at sed and awk (and sysadmin in general) you will not have (before you even had a job in IT) picked a singular task, studied it exclusively until you had complete command, then moved on the next discrete skill, studied it exclusively until you were an expert....etc. No one becomes an expert like that.

Well, you may want to reconsider that viewpoint, as that's exactly how a good number of network professionals obtain their skills.

They gather basic skills, then gain most of their actual expertise from using that skill in an appropriate production context to solve real problems.

That's all I am suggesting. I certainly never meant to imply that I or anyone should skip sed and awk skills entirely since it wasn't on the test.

To a degree. There are things you can learn in the Unix world that are essentially on demand. perl one-liners, some bash scripts, etc are like that. I'm of the bent that if something is worth learning, it's worth learning well, so if I encounter something I need to study in the midst of a project, I study it until I'm as much of an expert as I'm going to be. This usually involves a healthy amount of labbing so that I don't make rookie mistakes in production.

Trust me, you are not going to gain basic skills as a linux admin, and then all of a sudden rollout a multi-city mysql master-master replication setup over WAN links. You are going to do an awful lot of focused study, and you will hopefully do some lab mock deployments. If you're smart, your studies will include not just how to do it, but how to troubleshoot it and how to fix it when it breaks (because if there is one constant in IT, it is that it *will* break).

If you are seriously going to pursue a career as a unix admin, don't **** yourself. You've already got the book, work through it, it's worth it. Do yourself a favor and pick up the oreilly bash book and work through it too.

Again, I'm not trying to be a dick, it's just the way I communicate. I'm actually trying to help you!

I think this is the best approach, as we can't afford mistakes in production environments..thats what I did with most of the things I study..that's why I picked the RHCE eventhough it's not directly related to my job (to gain new skills that I don't usully encounter in my current job)...I'm gonna go and brush up on all topics but not in order and as time permits..specially that work keeps bugging me with new things..I will stop RHCE for the next month or so because work asked me to migrate production mail server from iplanet to the new Java messaging (pain).

Forsaken_GA wrote: »

... I'm of the bent that if something is worth learning, it's worth learning well, so if I encounter something I need to study in the midst of a project, I study it until I'm as much of an expert as I'm going to be. This usually involves a healthy amount of labbing so that I don't make rookie mistakes in production.

Trust me, you are not going to gain basic skills as a linux admin, and then all of a sudden rollout a multi-city mysql master-master replication setup over WAN links. You are going to do an awful lot of focused study, and you will hopefully do some lab mock deployments. If you're smart, your studies will include not just how to do it, but how to troubleshoot it and how to fix it when it breaks (because if there is one constant in IT, it is that it *will* break)..

Bl8ckr0uter

Forsaken_GA wrote: »

If you're smart, your studies will include not just how to do it, but how to troubleshoot it and how to fix it when it breaks (because if there is one constant in IT, it is that it *will* break).

Studying troubleshooting can be hard though.

The best way to learn to troubleshoot Linux is to make it your primary OS. Everything you normally do in Windows or Mac OS you should attempt to do in Linux instead. You will inject problems and break stuff on your own. You can then attempt to trace problems and resolve them. This is why the RHCE exam is so tough because many people believe they can walk in with little practical real world experience and pass it. If you do not use Red Hat, CentOS or Fedora regularly you will not be able to quickly find the answer to a problem on the test.

Chris:/* wrote: »

The best way to learn to troubleshoot Linux is to make it your primary OS. Everything you normally do in Windows or Mac OS you should attempt to do in Linux instead. You will inject problems and break stuff on your own. You can then attempt to trace problems and resolve them. This is why the RHCE exam is so tough because many people believe they can walk in with little practical real world experience and pass it. If you do not use Red Hat, CentOS or Fedora regularly you will not be able to quickly find the answer to a problem on the test.

Debian and Ubuntu are fine for learning Linux as well, you just have to do a little extra work to find a few things, and your package management is different. I always hate it when I have to go work on a RH box and it's like 'oh yeah the network config files are in a different place', but in general, if you can work with one form, you can stumble your way through another.

I do agree that immersion is the best way to learn it. only touching linux when you're studying, and then going back to doing whatever on your windows box is not a good way to make things stick