10+ year Linux systems admin/engineer, ask me anything!

Hey all, I'm Jeff! Found this site while searching for info on WGU, stayed after reading/answering a ton of threads!

I've been doing this stuff for over a decade now, and got my start in the field as a dial-up ISP support technician making 10 bucks an hour. I'm now a DevOps engineer on a cluster of Linux systems, I've added Apple, Citigroup, and Rackspace to the resume, passed the old RHEL 5 RHCT/E combined exam on the first try (and got my employer to pay for the bootcamp AND the exam), and I've navigated the job search and interview game to the point where the offer I accepted for my current gig was for $150k/yr, a percent or so of the company in pre-IPO shares, a $4,500 signing bonus, and a good benefits package...and if I paid attention to the job description/requirements, I'd have thought I was "underqualified" for it!

Why am I doing this?

I really enjoy mentoring other people in the field. Knowledge gained is knowledge that should be shared, in my opinion. Plus, one quirk of this field is that you never know who you'll end up working with, so it's in my best interest to make sure that the people I may be working with someday are rockstars!
I have a huge ego, and there's no better ego stroke than being asked for advice or to solve a hard problem.
I genuinely love everything about this field. It's not just my career, it's my passion. Plus, I get paid to solve puzzles and riddles for a living. How cool is that?!
We engineers regularly undervalue our skills, and since the vast majority of us like helping people and making things easier on them, we can screw ourselves when it comes to salary negotiation because we don't want to cause an imposition upon a potential employer.

So please, if you have a question on how to approach an interview or salary negotiation, please ask! If you have one on Linux systems design and troubleshooting, ask those too! If I know something about a topic, I'm happy to tell you what I know!

--

The Technomancer
"Any sufficiently advanced technology is indistinguishable from magic."
-- Arthur C. Clarke

Find more posts tagged with

Save $250 on 2025 certification boot camps from Infosec!

Book now with code EOY2025

Button

Comments

jamesleecoleman

What was it like when you started out at the Dialup ISP?
Do you have a Linux lab?

The Technomancer

jamesleecoleman wrote: »

What was it like when you started out at the Dialup ISP?

Pretty rough. DSL and Cable modems were starting to get popular and we were a full priced ($20/mo) dial-up provider in a NetZero/Juno world. We offered DSL service, but it never caught on. We were understaffed (30 minute wait times at one point), paid crap, and I worked nights on top of that, so I couldn't take advantage of mass transit and had to pay an hour of my pay every day to park.

I did learn a lot about troubleshooting in general, though. More than that, I picked up the ability to break down technical topics in to terms that a poor old lady out in the sticks could understand and follow. The best lesson I learned there was from one of the shift leads I worked under:

If you cannot explain something to someone and get them to understand it, that reflects far more on your mastery of the subject matter rather than the other person's intelligence and education.

It's served me well in my career. I've never been the best or smartest admin or engineer on my team. But I am the one that doesn't make the non-techies' eyes glaze over when I'm taking my turn presenting what I've done or learned at the end of the week, and that makes me the guy they want to work with when the fun project comes down the road.

Do you have a Linux lab?

I used to have a physical one. Then the wife got tired of the constant soft hum from the linen closet. Now it's virtual.

With AWS having a free tier, there's no reason whatsoever to not have a Linux lab. The only instance I pay for is the one I run my domain's mail server from. And if you actually need a lab to test issues specific to physical hardware, that's what dual-booting's for.

Run an RPM-based distro and a apt/dpkg-based distro. I suggest CentOS rather than Fedora for the RPM-based one, since it's rare even among startups to use the latest and greatest OS features or programs on the server side, so while Fedora's a preview of what's coming down the pipe, CentOS is a more accurate snapshot of what you'll encounter. For apt-based one, I'd suggest one of the LTS server versions of Ubuntu, since it looks to me like there's a shift among startups to using Ubuntu. You'll never get fired for choosing Debian, though.

The Technomancer

It's on my gaming rig because it's the best tool for that job, and despite it's flaws, no OS did more to bring computing to the masses than Windows. I throw Linux on my desktop every now and then to see how far it's come on the desktop, but I always go back to Windows.

Outside of Exchange for mail and calendaring though, I prefer Linux for all things servers. I'm biased, though!

I prefer doing work on Macs, though.

Bryzey

I am trying to start a Linux career in the next 6-12 months. Currently studying and working towards getting the skills to achieve my goal.

I am working on Linux+ at the moment then I plan to do two short courses on bash scripting and introduction to Python on Udemy before I start working towards RHCSA objectives in June 2014.

Does this sound like a good plan and/or should I include other things in my study plan before I start my RHCSA next June? Things like Nagios, SQL, SELinux, AWS I see in job ads all the time.

It is more about getting the skills over the certifications for me and getting my foot in the door once I feel like I have the skills.

Thanks.

-hype

Looking at your work history, how did you answer all the moving around you did? Looks like a new job every year.

Why did you leave Apple??

CCNTrainee

Where should I start if I want to learn Linux?? As I know to do very basic things, like install, run on a USB, Ifconfig, create user in CLI... that is about it. Lol

The Technomancer

Bryzey wrote: »

I am trying to start a Linux career in the next 6-12 months. Currently studying and working towards getting the skills to achieve my goal.

I am working on Linux+ at the moment then I plan to do two short courses on bash scripting and introduction to Python on Udemy before I start working towards RHCSA objectives in June 2014.

Does this sound like a good plan and/or should I include other things in my study plan before I start my RHCSA next June? Things like Nagios, SQL, SELinux, AWS I see in job ads all the time.

It is more about getting the skills over the certifications for me and getting my foot in the door once I feel like I have the skills.

Thanks.

You definitely want to learn BASH like the back of your hand -- The Linux Documentation Project has a great beginning and advanced BASH scripting guide that I've relied on ever since I found it. You'll also want to know one at least other scripting/high-level programming language. Ruby (thanks to Chef and Puppet) and Python are the big ones, with Perl and PHP still seeing a lot of use. I learned Python at Codecademy, but I hear Udamy's pretty good as well.

As far as certs vs. skills? The certs on the Linux end of things are great for picking up the basics, but outside of capstone ones like the RHCA, they're basically representative of a bare minimum of skill. The more you've touched and tinkered with, the better off you'll be because you'll be able to get a big picture view of how the systems work, and once you can design a cluster from that knowledge, congrats, you're an architect! The RHCE got Senior added to my title at the next gig I went to, but at the end of the day, being able to point to stuff you've done and projects you contributed to are better credentials than certification. Don't get me wrong though -- you'll never go wrong following a cert course to get an idea of the skills you'll be using on a daily basis.

The best advice I can give is come up with a project that sounds fun, and go to town on AWS's free tier (or drop a hundred bucks for a few instances there or somewhere else) trying to build it. You'll learn how to design the project or configure the service you want to provide, you're gonna break it a few times and learn how to fix it because of that (especially if you force yourself to fix it rather than slick the instance and start over), and you'll start adding man pages and reference sites to your bookmark list so you can do it faster the next time.

Once you know what to do as far as manually getting services going, then pick up one of the config management tools so you can do it across a cluster and have your boxes or instances boot up, get a base OS, get their config from Chef, Puppet, SaltStack, or Ansible, and check themselves into your load balancer and start taking traffic without you doing more than typing one command. If you're really good, your cluster will monitor itself and trigger new instances as needed without you doing a thing.

Do whatever it takes to get your foot in the door, and bust your ass once you're in. If the place you start at doesn't see your value, another company will and they'll pay you enough to lure you away. I can't speak for the Windows side of the house, but in the tech hubs, there aren't nearly enough Linux geeks to go around, so a bare minimum of skills, a boatload of curiosity, and a heaping helping of chutzpah will take you a long way and you'll learn a lot of neat stuff. IT is one of the last living meritocracies in this labor market -- take advantage of it!

Looking at your work history, how did you answer all the moving around you did? Looks like a new job every year.

Why did you leave Apple??

The only time I ever took a lateral transfer n my career was to go work on a video game, and getting my name in the credits of a video game with people whose names I recognized was a lifelong dream of mine. I had to leave Apple to follow it. I actually had a funny conversation about it with my wife when I was considering it -- Apple's a company you can retire from and retire well from. But she told me she didn't want to listen to me ***** and moan for the next who-knows-how-many years that I never took at shot at the video game company.

The fact that they were offering me a 30% raise didn't hurt, either.

When employers ask about my work history, I'm honest with them. I left Zix because I wanted a real sysadmin job, not a hybrid support/admin one, and they didn't have anywhere to promote me. ESPRE went bankrupt. I stayed at Rackspace for a bit over two years, got my RHCE on their dime, worked it off, and when they couldn't promote me, I took a consulting gig for Citigroup.

I left Citigroup because I had a perfect failover on one of the high frequency trading clusters on Wall Street to the DR site across the river in Jersey, zero downtime, no trades missed, but the added 7ms latency forced the trading team to switch to a different set of algorithms and on paper, cost the bank 1.6 million dollars a minute. Nothing like doing everything right and still getting cussed out on a bridge call. Plus, I wanted to settle in the SF Bay area, and once you tell that story, nobody asks you how you handle stress anymore.

On top of that, financial IT is a soul-draining experience. The pay is top notch, it's rarely going to work you more than 40-45 hours a week...but it's like living in Office Space. Mad props to the hardy few who can stick it out long term -- those dudes and dudettes are unflappable, and the unsung rockstars of the IT world. It damn near killed my passion for this craft. Believe it or not, employers like to hear that you know what you want out of a job, and if they can provide it, they'll offer it to you.

I left Apple to chase a dream, and I joined my current company when the dream didn't make much money and I got laid off along with a majority of the senior staff. Apple was even interviewing me to come back when the offer from my current employer ended my job hunt.

I also made sure I left every position with a glowing reference in my hand. It helps that I really love this stuff -- makes it easy to put in extra hours just by losing track of time because a tough problem really has my brain going. But even if your employer or your boss doesn't respect the hard work you put in, someone on your team will and they'll give you a reference when you start looking for another gig.

As long as you have the skills and the drive to back up the idea that no matter how long you stay with a prospective employer, they're going to get amazing work out of you and you'll leave them better off than when you joined, they'll overlook a non-traditional route that got you to where you are.

I'll also be the first to admit there's a lot of luck involved with taking the career path I did. People who graduate high school with a 2.27 GPA, drop out of college and don't get a cert until mid-career generally don't make it this far, especially this fast. On top of that, hard work won't take you everywhere, and anyone that tells you that is full of ****. Especially for us sysadmins -- if we're doing our job perfectly...nobody knows we're there because everything's running smoothly. You can literally automate yourself out of a job if you're not a credit whore (and this is why all IT minions are credit whores at heart, and should pretend to be if they aren't). What hard work does is allow you to take advantage of luck or an opportunity when it falls in your lap, and once you've worked enough gigs, you'll start to be able to spot opportunity. It's a skill like anything else, and it takes practice.

And for as rambling as that explanation was, that doesn't even really matter. Anytime an employer brings up a perceived flaw, whether it's job-hopping, inexperience, or anything else, they're implicitly devaluing your skill set, hoping you feel that it's something that needs to be defended rather than explained, and because of that, gain a bit of an advantage come offer time. Own what you've done, the challenges you've over come to get there, and speak in a manner that conveys the passion you feel for the job (and if you aren't feeling it, rethink your employment choices), and you start getting viewed as talent. Everyone wants their turn with talent, and they'll pay top dollar for it.

The Technomancer

CCNTrainee wrote: »

Where should I start if I want to learn Linux?? As I know to do very basic things, like install, run on a USB, Ifconfig, create user in CLI... that is about it. Lol

Congrats! You know more Linux than 90% of humanity.

Never undervalue your skill -- as long as you're realistic about what you can achieve with it, it's an asset and should be regarded as such. Nobody's going to regard it that way if you don't, though.

To get to the point of your question, though, pick something you want to learn how to do. Google how to do it. Then go fail at it. You will fail at it. If you didn't, you weren't curious enough about what all those settings in the howto do. Figure out why you failed. Get it working again. Start over and try to do it without screwing up the next time. Keep going until you've got it. Then figure out another project that sounds fun, and repeat!

Never be afraid to fail at something -- we all do it, and the best employers will ask you about your biggest screwup, what caused it, how you fixed it, and what processes or checks you put in place to prevent such a screwup again...and expect a good story rather than a sales pitch on how awesome you are. $DEITY knows I've made some spectacular screwups in my career. But you learn the most from those, if for no other reason than those screw ups strike the fear of $DEITY into you, and we humans tend to remember VERY VERY well how to avoid uncomfortable situations like that.

If nothing fun springs to mind, you're never going to go wrong knowing how to set up a server to run a LAMP (Linux for the OS, Apache for the web server, MySQL for the DB, and PHP for the code) stack, and tune it to run it fast, and there's a ton of well-documented howtos our there on how to do that along with popular variants of that setup. Learn BASH scripting -- everything because easier when you can sing to the OS rather than tripping over every third word.

The Linux+ and RHCSA courses are useful if you're a structured learner rather than an exploratory one (and both types can become rockstars, one ain't better than the other). They'll give you a solid foundation of tools, and should inspire a few projects to go work on -- if they don't, rethink taking the Linux route, because if you don't love this ****, you'll burn out spectacularly.

stryder144

The Technomancer wrote: »

If you cannot explain something to someone and get them to understand it, that reflects far more on your mastery of the subject matter rather than the other person's intelligence and education.

Amen and well-said.

Kinet1c

The Technomancer wrote: »

On top of that, financial IT is a soul-draining experience. The pay is top notch, it's rarely going to work you more than 40-45 hours a week...but it's like living in Office Space. Mad props to the hardy few who can stick it out long term -- those dudes and dudettes are unflappable, and the unsung rockstars of the IT world. It damn near killed my passion for this craft. Believe it or not, employers like to hear that you know what you want out of a job, and if they can provide it, they'll offer it to you.

Finally some credit! This will be my one and only job in the industry, never again.

What sort of linux specific questions would you ask an entry/junior level linux admin when interviewing them?

The Technomancer

REMOVED UNNECESSARY QUOTED REPLY FROM PREVIOUS POST

Phone screen:

List common ports and protocols (FTP, SSH, RDP, SMTP, POP, IMAP, HTTP, HTTPS, MySQL, DNS)
Are DNS requests via TCP or UDP?
Describe four types of DNS records for me.
Which umask gives files permissions of 0755 by default?
Which file, on a Red Hat-based distro, contains the networking information for eth0?
What do you enjoy about working with Linux?

Face to face:

Give me a rundown of init levels and what each one does.
Tell me what happens if you try to ping www . google . com and that address is not cached on your local machine or your primary/secondary DNS servers. Go into as much detail as you can.
I accidentally ran /bin/chmod -x /bin/chmod. Tell me why I suck for doing that, and tell me as many ways as you know how to fix it.
You have a standard three layer web cluster (web servers, cache layer, database). Users are complaining that the site is slow. Walk me through your troubleshooting steps. I will act as your Google/man pages for this exercise.
Tell me about the fix/hack you're most proud of.
Tell me about your biggest screwup, what caused it, how you fixed it, and what you learned from it and put in place to prevent it from happening again.

---

I prefer pepper jack cheese.

TheNewITGuy

So do you do anything other than Linux? Any cisco or MS admin

The Technomancer

I can do some basic stuff (VLAN config, tracking down what port a box is connected to, etc) on Cisco gear -- mostly the simple stuff so I don't have to open a ticket to a network admin over something stupid. I've had to admin Windows boxes before, and to be honest, I tossed Cygwin on them and treated them like Unix boxes until we brought a Windows admin aboard. I can troubleshoot services and such on them, but I don't have the knowledge to be a good Windows admin.

My secondary skills are programming and automation, which is why I gravitated towards the DevOps side of production engineering.

TheNewITGuy

Gotcha. Most of my roles are "heres this.. do it" - voice/data/windows/linux whatever haha

The Technomancer

A lot of my roles were like that as well early in my career -- but if I don't do stuff regularly, I forget the details, and most of my career was spent at the enterprise level where roles were pretty clearly defined. Now that I'm exploring startup land, I'm learning a lot of things really quickly because there is no network admin or phone guy or whatever to open a ticket to, and my boss is gonna tell me to RTFM if I ask him how to do it anyway, so I might as well RTFM and try not to break it much worse than it's already broken!

Using cloud services does help, though. I don't deal with a network outside of the local office, my PBX is a webpage in the cloud, my mail server is Rackspace's hosted exchange, Google apps and Atlassian Cloud act as my document sharing, ticketing system, and wiki, etc. It costs the company less to pay for those subscriptions than it does to pay me my hourly rate to build them in house and maintain them rather than working on our production systems.

JockVSJock

The Technomancer wrote: »

I throw Linux on my desktop every now and then to see how far it's come on the desktop, but I always go back to Windows.

I agree with this.

I've been running Linux in one distro or another since 2000 and while its made major progress on the desktop, MS Windows is de facto.

Plus, pretty much all major apps and web apps are made to run under Windows...so...as much as I love Linux, its a Windows world.

The Technomancer wrote: »

Do whatever it takes to get your foot in the door, and bust your ass once you're in. If the place you start at doesn't see your value, another company will and they'll pay you enough to lure you away.

Talk more about your experience where you walked from a job because they won't promote you, even though you are doing everything right. I've got certs, degrees and experience and I keep losing jobs to others who are less qualified.

I'm originally from Omaha, NE and there is small IT market there and I find people in their 50's and 60's doing desktop support and the only jobs are IT help desk/call center jobs.

jdancer

The Technomancer wrote: »

...
On top of that, financial IT is a soul-draining experience. The pay is top notch, it's rarely going to work you more than 40-45 hours a week...but it's like living in Office Space. Mad props to the hardy few who can stick it out long term -- those dudes and dudettes are unflappable, and the unsung rockstars of the IT world. It damn near killed my passion for this craft...

Haha, are you sure you didn't work side by side with me during my time in IT financial industry? Quoted for truth, brother. I used to sleep in the data center floor for the next trading day because I wanted to make sure there was no missed trades when doing maintenance during that window. I can still recall the shouts from the trade managers to the floor traders, "NOT ENOUGH EDGE F*KRS!"

The Technomancer

JockVSJock wrote: »

Talk more about your experience where you walked from a job because they won't promote you, even though you are doing everything right. I've got certs, degrees and experience and I keep losing jobs to others who are less qualified.

I'm originally from Omaha, NE and there is small IT market there and I find people in their 50's and 60's doing desktop support and the only jobs are IT help desk/call center jobs.

Once you're looking at moving out of entry level work to mid-level and senior jobs, your education and certifications are nothing more than a filter to get you the interview. Once you've gotten the interview, you need to talk about what you've done and how you approach your job more than going down a checklist of stuff you've memorized. Instead of talking about what you can do, talk about what you've done and talk about how it affected the big picture for your team, your department, and/or your company.

Working in a smaller or developing IT market has it's ups and downs. On one hand, if there's more work to go around than local talent, you can land jobs that you'd be underqualified for in a more developed labor market. On the other hand, it limits your options for workplace mobility -- if one or two shops know they're the only game in town, they're going to lag behind on pay and promotions. In my case, I moved away from SoCal, where the market was scarce in the early 2000s, to Texas where data centers and hosting companies were popping up due to cheap land and cheap power, and there was a local shortage of people, meaning that employers were paying well for the are and willing to train. My goal was to settle in the Silicon Valley/SF Bay area, and in order to command a salary to live there, I had to be very aggressive in chasing titles and a salary history to let me demand the senior gig and six-figure salary it takes to deal with $3k/mo rent, $4.50/gal gas, and other expenses that come with calling this place home.

In my case, I had a plan for my career, and if my current employer wasn't able to provide me the next steps towards it, I started to look for an employer that would. I've had job hunting rounds where I've not gotten offers from gigs I really wanted, and turned down offers for gigs that paid well but wouldn't teach me anything (there's truth to the phrase "trapped by money"), and gigs where I'd learn a ton but wouldn't pay me what I was worth. Keep your big picture goals in mind when accepting a position, and never plan to retire from someplace. If you do find a place like that to stay for decades, awesome! But at the end of the day, the employment relationship should be one where both parties profit from the partnership -- you gain more skills and see how another shop does things, and they make a profit off of your skills getting applied to their business.

The final piece of advice I can give is that if you're in a position to be picky about job offers, be picky. Find an employer that's going to give you as much time as they can working on projects that are your passion. Don't be afraid to do the crap work like documentation and in-the-weeds troubleshooting, but be insistent on getting time for projects that'll expand your skillset, and if the employer can't provide that, slog through your year and start searching for something else. Nobody puts in their best work on stuff that bores them or they dislike, no matter how hard they try. Put yourself in the best possible situation to proceed, and be sure to choose an employer that's going to facilitate that if you earn it.

EDIT: Oh, be sure you're tooting your own horn every now and then. Again, if we're doing our jobs well, nobody sees the fruits of our labor. Take credit when it's due, share credit to your team when applicable, and be proud of what you've done, no matter how insignificant it may seem.

JDMurray

The Technomancer wrote: »

In my case, I moved away from SoCal, where the market was scarce in the early 2000s,

Where were you in SoCal? I'm in Irvine--the Silicon Valley of The OC.

The Technomancer

North LA county -- not just a real desert, but a technical one as well if you aren't military!

Slowhand

Seeing as this is a forum dedicated to IT certifications, I'd be interested in hearing your take on Linux-related certifications and their place in the industry. Obviously, Red Hat's credentials carry a lot of weight in most places, but what about things like LPIC, Linux+, Novell's certifications, etc. ? What has your experience been in talking to coworkers, seeking jobs, and generally being around other Linux-focused professionals?

As someone from the Microsoft-oriented side of the house, where certifications and specializations do make a difference when it comes to upward mobility and actually finding new work, I'm always curious about what the landscape looks like in other parts of the IT field.

The Technomancer

Slowhand wrote: »

Seeing as this is a forum dedicated to IT certifications, I'd be interested in hearing your take on Linux-related certifications and their place in the industry. Obviously, Red Hat's credentials carry a lot of weight in most places, but what about things like LPIC, Linux+, Novell's certifications, etc. ? What has your experience been in talking to coworkers, seeking jobs, and generally being around other Linux-focused professionals?

Let me preface my comments with the fact that these are my opinions I've formed from my experiences, and therefore, YMMV and load up on salt grains and such.

Linux+: It's a great structured course and cert for someone that's hardly into computing in the first place, let alone Linux, and makes a great entry point for someone that doesn't know where to start learning at. Beyond that though...when I worked with Alamo City Colleges to design a 24 hour course that'd teach someone the basics they need to know to become a sysadmin, I took the 40ish hours of Linux+ material, chopped out the historical and hardware-related stuff that wasn't Linux specific, ended up with 20 hours of material, then added 4 hours on common services like Apache and setting up MySQL. Don't expect people to give a damn about it, though, unless you're trying to break in the door, at which point the hiring manager can at least be assured that you aren't going to look at a command line and go "WAT IS THIS I DON'T EVEN".

RHCSA: I took my RHCE when this tier was the RHCT and it was a combined exam with the RHCE, but I've heard pretty good things about it, and the crowd I run with doesn't always have the highest opinion of certs.

RHCE: The gold standard cert for a junior sysadmin to prove they're ready for a full grade admin gig. You don't use half the stuff in there (SELinux contexts are great to know if you're working for the government, but everyone else turns SELinux off), I've never actually used extended file ACLs while doing real-world work, and xinetd can go to hell -- use supervisor if you must use a superserver rather than managing your services individually via a config management stack like Puppet. The rest of the stuff you're taught with the RHCE materials IS real-world applicable.

Upper level Red Hat certs: Nobody cares until you complete enough for the RHCA. Employers that do care will hire you as an RHCE and send you for the ones they want.

LPIC series: Just as good at the Red Hat ones, but they aren't the hot name, so if you only had money/time for the RHC* series and the LPIC series, choose the RHC* ones.

Novell's: I haven't worked in a SUSE shop, so I don't have enough experience to know if Novell's certs are good or not.

As far as renewing them? If you have an employer that'll pay for it (or pay you for keeping them current), go for it. Otherwise, once you've hit the 5 years experience mark, people care more about what you've done and how you approach abstract problems than your degree or your certs. At the 10 years experience mark, if they still care about them, they're the government, they advertise the fact that their employees are certified because it sells more widgets, or you don't want to work there.

My RHCE gets end-of-life'd 6 months after RHEL 7 hits (Mine's for RHEL 5 and grandfathered in under the old rules, not the new 3 year limit), and I don't plan on renewing it unless my employer at the time wants to pay for it -- but I'll also qualify that with the fact that I don't apply to jobs anymore, employers headhunt me, so it's entirely possible I don't have a full picture of what those certs do anymore at the midlevels.

As someone from the Microsoft-oriented side of the house, where certifications and specializations do make a difference when it comes to upward mobility and actually finding new work, I'm always curious about what the landscape looks like in other parts of the IT field.

Certs like the Linux+ and RHCSA will help you break in with established companies when you're trying to get your foot in the door, no question about that. The RHCE (and LPIC II, for shops that recognize its value) will speed up your transition from top-tier support or Jr. Admin to full-grade sysadmin, and an RHCA will never lack of opportunity or job offers.

At the end of the day though, it really depends on what kind of company you want to work for. If you want a low-risk, rarely over 40 hours a week, stable enterprise-level gigs that'll give you a comfortable middle-class life and retirement, these places like certs and like sending people to training for them, and you should definitely cert up and take every training opportunity afforded to you. You'll also never make the top dollars available, but this is the difference between doing this for work and doing this because you can't see yourself doing anything else.

If you're looking at working for startups or firms on the bleeding edge of tech, or you're looking to become an expert consultant to come solve problems for companies that their guys can't fix, certs don't do you much good because what you're being taught isn't what the employer is using. It's all about what you've done, how much theory you know, how well you can figure out issues and problems given a limited set of information, and building those systems in such a way that it doesn't all crash and burn under the next traffic spike while you're on a shoestring budget. The education needed to receive a cert definitely sets you up with a set of tools to draw from, but you have to be using this stuff on a regular basis and exploring the boundaries of what's possible with not just the tools you're taught, but the ones that may be better suited to what you're planning to do that'll give you just enough of an edge to beat out the other 547682948765 startups trying to do what you do.

No matter which route you choose, though, one of the things that drew me to Linux was the open source aspect. Not because I particularly give a crap about the whole free software/open source movement (although it does hold a place in my shriveled, blackened BOFH heart), but because it opens up an opportunity to make a name for yourself by contributing to projects that admins everywhere use and allows you to short-circuit the normal career ladder with enough time and work put in.

Slowhand

I'm not quite sure I'd agree with your assessment that certifications - and other industry credentials - are effectively useless after your first five years in the industry. That may be the case with *NIX admins, or perhaps it's been your particular experience, but there certainly are other scenarios that have been discussed at length here on TechExams:

In some cases, sitting with an expired cert from several years back can make an IT professional seem like they were pigeonholed, like they never moved beyond that one particular technology or job role. This perception can be very powerful, especially if you're thinking of switching jobs or applying for a promotion within your own company. Appearing to be an expert on obsolete technology can, in some cases, be as damning as having no credentials to show at all. This isn't, of course, set in stone or the way every employer or manager sees things, but I've certainly come across this attitude in the wild.

Beyond the outside perception, it's difficult to keep up to date on broad, sweeping technologies simply based on what you're working with. A focused curriculum, like a certification-path, makes it easier to "force" you to read up on what's new and what's changed since the last time you really had to learn about this stuff. It's also easier to digest new changes on the whole, when you're given the big picture rather than stumbling across it here and there. We all remember how oddball virtualization seemed when it first became a buzz term in the industry. Without the more robust reading list I have for, say, the VCP course I'm working my through at the moment, I very much doubt I'd be as familiar with virtualization on the whole, rather than the few moving parts I've touched while working. I also don't think I'd be able to do much more than what I saw through that narrow window of past experience, without proper training and instruction in how to best take advantage of this (now not so much) new technology. And nothing quite lights a fire under your butt to get training done like having something to show for it at the end.

Differing opinions aside, I'm sure that others who are following in your footsteps of working as a Linux sysadmin appreciate any advice and wisdom you have to share in that field, particularly here in the San Francisco Bay Area where the market for IT folks with open-source experience is particularly competitive. TechExams is always happy to have another voice to join the discussions here on our forums.

The Technomancer

Slowhand wrote: »

I'm not quite sure I'd agree with your assessment that certifications - and other industry credentials - are effectively useless after your first five years in the industry. That may be the case with *NIX admins, or perhaps it's been your particular experience, but there certainly are other scenarios that have been discussed at length here on TechExams:

If I wasn't clear, I was referring to *NIX admins only, and those working outside of larger institutions where a portfolio of increasingly advanced certifications can prove to be a boon to one's career. I'm definitely not trying to devalue the worth of certifications and education, only pointing out that depending on what type of *NIX role you're looking for, they may not get you there.

The RHCE was instrumental in getting me the consulting gig with Citigroup. I have no doubt that I wouldn't have gotten the interview without that on my resume. I've just moved past the point in my career where the RHCE is still particularly useful unless I want to go all the way for the RHCA. If I planned to go back to Enterprise, finance, or some other larger shop? I'd re-up, no questions asked.

In some cases, sitting with an expired cert from several years back can make an IT professional seem like they were pigeonholed, like they never moved beyond that one particular technology or job role. This perception can be very powerful, especially if you're thinking of switching jobs or applying for a promotion within your own company. Appearing to be an expert on obsolete technology can, in some cases, be as damning as having no credentials to show at all. This isn't, of course, set in stone or the way every employer or manager sees things, but I've certainly come across this attitude in the wild.

No doubt. If you aren't able to show career growth in other terms, like increasingly important titles and responsibilities, you should definitely get certifications (and re-up the ones you have) as proof that you don't have a dusty skillset.

Beyond the outside perception, it's difficult to keep up to date on broad, sweeping technologies simply based on what you're working with. A focused curriculum, like a certification-path, makes it easier to "force" you to read up on what's new and what's changed since the last time you really had to learn about this stuff. It's also easier to digest new changes on the whole, when you're given the big picture rather than stumbling across it here and there. We all remember how oddball virtualization seemed when it first became a buzz term in the industry. Without the more robust reading list I have for, say, the VCP course I'm working my through at the moment, I very much doubt I'd be as familiar with virtualization on the whole, rather than the few moving parts I've touched while working. I also don't think I'd be able to do much more than what I saw through that narrow window of past experience, without proper training and instruction in how to best take advantage of this (now not so much) new technology. And nothing quite lights a fire under your butt to get training done like having something to show for it at the end.

I think our opinions on this differ because I suck at classroom learning or trying to absorb lessons from a book. If I can't get my hands on it and play with it, I don't learn it, and I consider this a weakness on my end. I've just learned to adapt around it by finding jobs that'll let me play with the gear or tech I want to learn so I can master it.

Certs might not be for me, but I truly hope that nothing I've said has discouraged people from obtaining them. Do what works for you, because you gotta live your life and you hopefully have an idea of how you learn best and what makes you a success. Certs will get you a strong base of knowledge and a phone screen. However you acquire that base of knowledge and tools in your toolbelt, abstracting the lessons learned from them and applying them to tech or builds outside of the technologies and methods those credentials certify is what will land you the job, especially at the upper levels of seniority.

Differing opinions aside, I'm sure that others who are following in your footsteps of working as a Linux sysadmin appreciate any advice and wisdom you have to share in that field, particularly here in the San Francisco Bay Area where the market for IT folks with open-source experience is particularly competitive. TechExams is always happy to have another voice to join the discussions here on our forums.

I've found the people here really welcoming, and I hope to stick around for a while! Given that you're in the Bay as well, we ought to catch a beer sometime. Are there any local TechExams drink-ups/meetups or anything?

Krones

I am interested in devops. I know my weak spots. Linux and Python. Pretty big holes for any aspiring devops engineer. I am stuck in the trenches. I get to install basic centos and configure raid arrays and install xen server but other that I am not getting much hands on experience on our production environment. I am pretty much IT Grunt/helpdesk on the Windows admin side of the house. The main hold up are my classes at WGU which prevents me from going full bore on my linux/python skill building outside of work.

What five books or resources would you recommend for any aspiring devops system administrator. Outside of Linux or Python books or one of each.

We run gearman, haproxy, mysql, cassandra, memcache, Redis, Sphinx etc, and other scalable open source technologies. What open source technologies should I be looking into that you find fascinating.

One decent resource I have come across is: Ops School Curriculum — Ops School Curriculum 0.1 documentation

Jackace

The Technomancer wrote: »

Once you know what to do as far as manually getting services going, then pick up one of the config management tools so you can do it across a cluster and have your boxes or instances boot up, get a base OS, get their config from Chef, Puppet, SaltStack, or Ansible, and check themselves into your load balancer and start taking traffic without you doing more than typing one command. If you're really good, your cluster will monitor itself and trigger new instances as needed without you doing a thing.

Any good tutorials or resources for how to do this in a Home Lab setting? In particular the load balancing part. Do you just run a linux VM that does the load balancing or do you use something like an F5?

The Technomancer

Krones wrote: »

I am interested in devops. I know my weak spots. Linux and Python. Pretty big holes for any aspiring devops engineer. I am stuck in the trenches. I get to install basic centos and configure raid arrays and install xen server but other that I am not getting much hands on experience on our production environment. I am pretty much IT Grunt/helpdesk on the Windows admin side of the house. The main hold up are my classes at WGU which prevents me from going full bore on my linux/python skill building outside of work.

What five books or resources would you recommend for any aspiring devops system administrator. Outside of Linux or Python books or one of each.

Pulling Strings with Puppet: Configuration Management Made Easy, James Turnbull. You're going to need to know at least one config management system like the back of your hand. Right now, Puppet and Chef run the show. I know Puppet, so I'm recommending a Puppet book, but I'm sure there's quality Chef texts out there as well.

Systems Performance: Enterprise and the Cloud, Brendan Gregg. This book picks up application and performance troubleshooting where Google and StackExchange stop being useful on a regular basis. The sample chapter available on the net is on CPUs, and the entire book is of that quality.

Learn to code | Codecademy <--- I learned Python here.

The Linux Documentation Project <--- I learned BASH here from their beginners and advanced scripting guide.

Outside of that, when you want to learn a certain application or service, you generally won't go wrong by purchasing an O'Reilly book on the topic.

We run gearman, haproxy, mysql, cassandra, memcache, Redis, Sphinx etc, and other scalable open source technologies. What open source technologies should I be looking into that you find fascinating.

One decent resource I have come across is: Ops School Curriculum -- Ops School Curriculum 0.1 documentation

I'm liking Nginx far more than Apache nowadays for web services. It's a bit harder to configure, but outperforms Apache in nearly all cases. For physical hardware/PXE boot environments, Cobbler's pretty awesome, especially since you can set up pre-install and post-install triggers to fire off your config management software and run scripts to check your box into a load balancer from there.

I'm also pretty fascinated with using custom-built APIs to communicate with my servers -- saves me having to deal with a parallel SSH client or something like Fabric to run commands and scripts across an entire cluster, and it makes it very easy to poll for statistics when I just have to curl a GET on $hostname/stats/cpu to get a snapshot of CPU usage. It's a ton of work up front to get it all in place, but it basically takes care of the day-to-day admin work on it's own and I can focus on the fun parts of the job.

@Jackace:

You can run something like Zeus or haproxy as a software load balancer. If your lab's virtual in something like AWS or Rackspace Cloud, you can use their APIs for Elastic Load Balancing or Cloud Load Balancers to let your boxes check in with them.

There's no real tutorials on how to do it (that I found, at least, but I didn't look that hard, to be fair), other than reading the API documentation, and making the final action your config management software takes on first run the launching of a script that interacts with the API to check the server/instance in with the load balancer.

ally_uk

Fantastic thread

I work in a Windows environment. But have developed a obsession / passion about Linux I'm constantly reading books up different subjects, I guess for myself it's a hobby my colleagues switch off and I hear the normal cliche Linux is not relevant arguments. But the passion for myself has continues. My current skill level I can use the Command line and use tools like Grep, install packages, use tar and rsync I have dabbled with servers at a basic level got a samba server up and running nothing spectacular real basic level stuff. I have now veered off down the Bash Scripting route, I have created basic scripts, You know the noob echo and variable declaration stuff and chaining commands together. Question is I have touched on command operators the more advanced stuff like loops and am getting a bit confused. So the question I ask you is what is the best approach to break down this subject any recommendations for resources?
Like I said I'm passionate about this and can't wait to get my hands dirty

The Technomancer

@ally_uk:

Depends entirely where you're getting confused.

For loops take a list provided to them and do something to each item in it.
While loops do whatever is in them while a statement evaluates to true.
Until loops do something until a statement evaluates to true...the opposite of a while loop, basically.

BASH Programming - Introduction HOW-TO: Loops for, while and until has a great explanation, along with examples.

In fact, start here, then continue to here.

If you're having a specific problem, I'll be happy to help!

jem7sk

The Technomancer, where were you 15 years ago when I first got into IT. I have only known one other person in my career close to your level of knowledge but he was a jerk and didn't like to share. Today I work for a large school district on a windows domain and there is no one that could touch your knowledge... Far from it. I have always wanted to get into the linux side of system admin like you. If I am reading correctly what you are suggesting... I need to get Centos: learn bash, python and puppet and I will be well on my way?

Also, what forums, web sites and other resources do you use to learn about new technology to stay updated on your line of work?