10+ year Linux systems admin/engineer, ask me anything!

jem7sk

The Technomancer, where were you 15 years ago when I first got into IT. I have only known one other person in my career close to your level of knowledge but he was a jerk and didn't like to share. Today I work for a large school district on a windows domain and there is no one that could touch your knowledge... Far from it. I have always wanted to get into the linux side of system admin like you. If I am reading correctly what you are suggesting... I need to get Centos: learn bash, python and puppet and I will be well on my way?

Also, what forums, web sites and other resources do you use to learn about new technology to stay updated on your line of work?

The Technomancer

CentOS is a good choice to learn on.

I'd tackle things to learn in this order:

Installing and configuring common services like Apache and MySQL. Look at job openings to see what the business in your area are asking for.
Learn to fix those services when they break. If you aren't breaking them, you aren't playing with them hard enough.
Learn how to tune your server to get the maximum performance out of your servers...or at least find out where to look for the common tweaks.
Learn BASH to automate the common tasks those services require, like log parsing or load testing...if you haven't already gotten tired of doing all the previous items manually and started looking at ways to make it easier.
Learn to install and configure a monitoring suite like Nagios (or one of it's forks), and a secondary, statistics collecting/displaying one like Cacti, or the stats/Graphite combo.
Learn a config management package like Chef or Puppet (SaltStack's fun, but it'll be a few years before it's common) to install those servers and place their configs and your scripts across a bunch of servers at once.
Learn Python or Ruby so can start building your own tools when the ones made by other people just don't start cutting it anymore, and to do the bigger tasks that BASH just isn't cut out for (you'll find them, trust me).


To stay up to date, I try to attend a few conferences a year. I try to go on one interview a month to see what's being asked in them, and I keep an eye on the job boards to see which technologies are popping up more often -- and if it's one I don't know, I'll start playing with it. I keep an eye on the StackExchange sites, Slashdot, and lwn.net, and I'm subscribed to a few Linux users/kernel mailing lists.

Master Of Puppets

How do you feel about antivirus software on Linux and do you use any? If you do, any recommendations? Of course, Linux isn't as vulnerable as Windows but there are some treats for it too. Personally, I think it doesn't hurt to have one installed.

The Technomancer

It's a waste of disk space, memory, and CPU cycles on anything but a mail server, and you're putting it there for the purpose of scanning email for viruses and removing them, not to protect the server itself. Verifying the checksum of software about to be installed ensures you're getting what you thnk you are, and any software you aren't sure about either shouldn't be run until the source can be inspected or can be run in a chroot jail or virtual machine and strace'd to determine all the actions it's taking -- plus, in a professional environment, that software's going to be deployed at least twice before ever hitting production (and this is why your dev and staging environments are on their own VLANs and ACL'ed off from any servers that matter) and if you have the monitoring going on your environment that you should, you'll notice the uptick in resource usage well before malware hits production.

If you're installing unverifiable binary blobs onto a Linux server, you're wrong. If you have to be wrong and you install it straight to servers that matter rather than a lab or VM environment first (or at the absolute minimum, a chroot jail), an anti-virus suite ain't gonna save your career from the fact that you're disregarding best practices and seeing solutions where there aren't reasonable ones (because if you can't test and verify the output, it's not a reasonable solution).

xenodamus

Nice thread - glad to have you with us!

I've actually been plotting my entrance into Linux administration. I've worked as a junior Windows and Network engineer in the past and I feel like I need to dive into Linux to round out my skillset and decide once and for all which direction I want to go. I work in an environment with lots of technology, and not enough people to run it.....which means open season for anyone willing to learn! Sometimes I feel like I have IT-ADHD . You've given me a lot of good info to get rolling with, though.

Have you always been Linux focused? How did you come to settle on that as your path?

The Technomancer

I've always been Linux-focused in my systems-related work. The dial-up tech support job that allowed me to get my foot in the door in the industry did teach me some desktop Windows troubleshooting, but I haven't done it professionally since then.

I actually got my start in Linux because I wanted to **** at an online video game. You had to set up a Linux box to run the **** app as they kept it on Linux to keep it low profile. By the time I got it all working and running, I found out that I enjoyed playing around with this OS more than I enjoyed playing the game I was trying to **** at!

I managed to turn a hobby into a career, and I'm loving (almost!) every minute of it.

UnixGuy

Thanks for this thread, great points so far.

What's your experience with Storage (SAN, NAS) arrays? What's your experience with Enterprise Backup software (Netbackp, Legato, TSM) ? What's your experience with Linux HPC solutions and distributed filesystems? I'm just curious to know with 10 years of experience, what do you expect to know about those technologies...

and finally, what's your thoughts about Unix (Solaris, HP UX, IBM AIX) ? Do you see them often in the market? Where do you think they fit?

The Technomancer

UnixGuy wrote: »

Thanks for this thread, great points so far.

What's your experience with Storage (SAN, NAS) arrays?

Used NetApps in a few places in my career thus far, along with stuff like MSAs and homebrewed shove-a-bunch-of-disks-in-a-box-in-RAID-whatever-and-share-it-via-NFS "solutions". NetApp's got great tools and there's plenty of **** sheets out there.

What's your experience with Enterprise Backup software (Netbackp, Legato, TSM)?

I've used NetBackup, but hated it since it was pulling from a robotic tape library. Start a restore when you leave for the day, come back the next and it might have loaded by then! Bare metal restore's pretty cool, though.

Nowadays, I don't use it much, since I only really have a few important servers that have data that needs backup outside of regular off-site syncing. Everything else isn't important or goes in Puppet, which gets checked in to Git and backed up off site.

What's your experience with Linux HPC solutions and distributed filesystems?

I've had to admin Hadoop servers, and I've used HDFS and MongoDB's GridFS for distributed filesystems.

I'm just curious to know with 10 years of experience, what do you expect to know about those technologies...

Depends entirely on what your focus has been. You see a lot of these techs in large enterprises. And really, the only way you get experience on them is to get to play with them, since you're not gonna drop a quarter of a million on a NetApp for a lab, and few colleges are going to either (at least, for systems they actually let the students touch rather than the researchers), and most of the other software is also out of reach for the person that wants to tinker.

and finally, what's your thoughts about Unix (Solaris, HP UX, IBM AIX)?

I miss dtrace when not on them and I have to troubleshoot something crazy.

Do you see them often in the market? Where do you think they fit?

Finance and government still uses a lot of AIX and Oracle, especially if they have/had a contract with IBM or Sun/Oracle. While I don't have direct experience in government work, I assume they'd be in heavy use there in older departments. I haven't seen HP-UX for a while, but I also don't tend to look for jobs in sectors where it might be used, so I'm unfortunately not a good source for info on that one.

I think as long as the OS is the right tool for the job, it should be used, and as long as that's the case when it comes to a true Unix over Linux, there will be a place for it. There will be legacy support work for decades beyond that. And frankly, if you can bend Unix to your will, you can land a job as a Linux admin if the work ever dries up -- there's a ton of "<flavor of Unix> to <flavor of Linux> **** sheets. No dtrace, though. =(

In startup land, it's Linux by a long shot, with a smattering of FreeBSD.

UnixGuy

Thanks for the answers. Yeah true that it depends on the job. Some places expects you to have experience with EMC SAN Arrays (for example), and a backup software or two.

The Technomancer

While it may be the function of how hot the market is here in SF/SV, requirements in job postings for Linux/Unix work out here tends to actually mean "Please have more than half of these", and preferred experience or nice-to-haves mean "Please have heard of these, or something like them." If your core skillset is solid, employers will train you how to use the enterprise gear and software that only gets purchased by employers that are that size, and if you can logic out a solution to a problem or design a system to overcome it, employers will teach you the tools necessary to make those solutions happen.

UnixGuy

True. It's like that everywhere

jem7sk

The Technomancer wrote: »

To stay up to date, I try to attend a few conferences a year. I try to go on one interview a month to see what's being asked in them, and I keep an eye on the job boards to see which technologies are popping up more often -- and if it's one I don't know, I'll start playing with it. I keep an eye on the StackExchange sites, Slashdot, and lwn.net, and I'm subscribed to a few Linux users/kernel mailing lists.

Technomancer, Thanks for this and everything else you posted in regard to my question above. Just curious which job boards you watch?

EngRob

The Technomancer wrote: »

I actually got my start in Linux because I wanted to **** at an online video game. You had to set up a Linux box to run the **** app as they kept it on Linux to keep it low profile. By the time I got it all working and running, I found out that I enjoyed playing around with this OS more than I enjoyed playing the game I was trying to **** at!

Great thread! I have to ask....what online video game?

The Technomancer

@jem7sk:

Dice is good. Sites for recruiters that work with companies in your area will usually have jobs listed that they're recruiting for as well. On top of that, keeping an up to date LinkedIn profile should get you recruiter spam, and those will have job descriptions listed.

@EngRob:

Everquest.

rsutton

Did I hear something about a Bay Area meetup involving beer? Interested!

The Technomancer

Man, after this week, I really need a beer or 6. Boss spent the last week in Hawaii, and most of this week at AWS's re:Invent conference, and of course, everything catches on fire.

I've had to babysit or manually run a bunch of data ingestion jobs, every deploy went to crap (at least it seemed like it), account managers had me chasing bugs in the releases (because the lead devs were off at re:Invent as well!) and the little free time I did get to breathe I've either posted something here or worked on the API I'm writing to replace the collection of BASH scripts with terrible code smell that the dude I replaced left behind as "automation".

I might get out of here before midnight. Oy.

W Stewart

How do you feel about arch linux and gentoo and what factors do you feel has contributed to your success in your career field?

The Technomancer

Arch and Gentoo are fun to play with, but Gentoo enthusiasts make my brain hurt. I wouldn't put either of them in production, especially given the Gentoo community's amazing ability to publish patches to bugs that don't exist (or have already been fixed upstream).

Things that have helped me find success:

* Making long-term goals to shoot for, then breaking them down into shorter term goals that will lead me there.
* Being willing to look for a new job if I'm not accomplishing my goals at the place where I'm currently working.
* Absolutely loving what I do
* Working hard enough to take advantage of luck/opportunity when it comes up
* Learning how to manage my manager
* Being able to admit what I don't know
* Having soft skills

Bryzey

From what I see on job ads here in Australia there is either big corporate type companies or Web hosting companies advertising Linux roles.

What sort of difference in the work is there?

Seems like 2 totally different worlds yet they ask for the same skills. 5 years experience, a scripting language and strong Linux skills along with a few other things on automation, networking, monitoring.

Also do you think one is better then the other as a starting point?

The Technomancer

Big corporations, if they're not an Internet-focused company, will probably have you managing their internal infrastructure, storage, and stuff like that. It's a valuable way to gain experience on enterprise-class gear. Web hosting firms will teach you to master the LAMP stack, and you'll get to touch a lot of bleeding edge tech since smaller customers will be demanding it to try to get an edge on more established competition.

I did over two years in web hosting, and I couldn't recommend it enough. It's not a fun job, but you will get to touch a lot of things, learn how to tune servers to serve content faster than normal. Both choices, however, will get you valuable experience and set you up for a senior role later on down the road.

If you want to share what excites you about Linux systems administration, I may be able to offer better advice, since both types of companies are going to expose you to very different things. They use the same fundamentals, but the applications you'll be administering will likely be very different.

Bryzey

I'm yet to get any on the job exposure but from my studies so far I have enjoyed the Web stuff.

W Stewart

Do you have a degree and do you feel like not having a degree would be detrimental to ones career as a linux administrator?

varelg

I plan on taking RHCSA and RHCE, have no experience in IT and plan to go independent after getting those certs. What work should I expect coming my way, if any?

Tremie24

What's your career path like? From entry level to now. I'm still trying to land my first IT gig, but Linux is one of paths I'd like to pursue. Any advice on what type of entry level jobs I should be looking for?

The Technomancer

Sorry for the late responses! Been hectic trying to get work done before the holiday.

Bryzey wrote: »

I'm yet to get any on the job exposure but from my studies so far I have enjoyed the Web stuff.

You'll probably want to go work for a smaller shop then. You'll learn good fundamentals there without the aid of enterprise-class tools, as well.

W Stewart wrote:

Do you have a degree and do you feel like not having a degree would be detrimental to ones career as a linux administrator?

I'm a college dropout, and I was on track for a Political Science major with a pre-law focus. On the Linux side of the house, not having a degree doesn't really hurt you since most college programs aren't teaching you the tools that Linux admins often use. Google turned me down back in the day over not having one, but that's the only time it's ever mattered, and now Google's reached out to me a few times since I've gotten more experience.

Outside of a few academically-focused firms like Google (or government work, where they aren't as lenient about replacing a degree requirement with work experience), a college degree is, at best, a replacement for 4 to 5 years of work experience insofar as landing an interview goes. Having the fundamentals you learn about networking and computing from an MIS or CS/CE degree is always helpful. If you can afford it without going into debt, I'd recommend college. If you learn best in a structured environment rather than under deadline pressure, I'd recommend college. I just wouldn't go into debt over it if you want to do Linux work. There's way too many other ways to gain experience without college.

varelg wrote:

I plan on taking RHCSA and RHCE, have no experience in IT and plan to go independent after getting those certs. What work should I expect coming my way, if any?

You'll probably be able to land some mid-level systems administration consulting work on a contract basis, but depending on your market, the competition may be tough. I've mentioned the 5 year and 10 year experience levels -- those really matter on the Linux side of the house. Those two resume markers will do more for your career than any certification or training outside of a capstone certification like the RHCA. If you want to go independent ASAP, I'd really recommend investing the $40k or so that it costs to take training classes and exams all the way to the RHCA. Otherwise, I'd get the RHCE, go join a firm that has a history of paying to certify its employees, and suck up working for someone else until a year after they've paid for the RHCA.

Tremie24 wrote:

What's your career path like? From entry level to now. I'm still trying to land my first IT gig, but Linux is one of paths I'd like to pursue. Any advice on what type of entry level jobs I should be looking for?

Working in a data center (if you can lift 50 pounds) for a hosting company or on their entry level support team is likely to be one of the best entry level gigs you can get for learning Linux. Networking (the people kind) matters a lot -- got my first systems admin gig via a referral from a friend. Check to see if there's any Linux user group meetups or anything like that in your area. If not, start one. Meet the Linux professionals and enthusiasts in your area, and build relationships. You meet interesting people, and get interviews for jobs that never get listed.

My career path + salary history:

Jan. 2001- Jun. 2003: Tech support for a dial up ISP. $10/hr, Dallas, TX.
Jul. 2003 - Aug. 2007: Ran my own email and web hosting company while trying out college. Learned a lot about how not to run a business. Lost money on the venture, but managed to not run up very much student loan debt.
Aug. 2007 - Aug. 2008: Systems admin/support tech for an email encryption company. Referred by a good friend. $41k/yr salary, Dallas, TX.
Aug. 2008 - Feb. 2009: Consulting systems administrator for a mobile video startup. It bankrupted. Whee! $24/hr, Plano, TX.
Feb. 2009 - May 2010: Linux Systems Admin II for Rackspace. They paid for my RHCE. $49k/yr salary, San Antonio, TX.
May 2010 - Jan. 2011: Senior Unix Systems Administrator for Citigroup. Consulting gig. $37/hr, Irving, TX.
Jan. 2011 - May 2012: Senior Systems Engineer for Apple. $100k/yr salary, stock package worth about $60k, paid relocation from TX. Cupertino, CA
May 2012 - Sep. 2013: Senior Systems Engineer for a video game company. $130k/yr salary, equity package likely to be worth zilch. San Mateo, CA.
Sep. 2013 - Present: DevOps Engineer for a big data/ad firm. $150k/yr salary, equity package that I'm busting my ass off to make worth a lot. San Francisco, CA.

linuxlover

Hi Technomancer,
I'm so glad you started this thread. I want to work with Linux for the rest of my life hence the certification path (so that I could find my first IT job). I have no official work experience other than running my own free web hosting service but I feel like I'm not doing all that stuff you do in a big company. Actually I don't do much if anything at all, because it all runs by itself. So my question to you is, what books would you recommend to learn Linux and its components the geek-way? I'm not talking about standard books like Linux Bible, but those kinds of books that are really valuable but hidden from noobs, because we don't know what to look for.

You mentioned two books previously: Pulling strings with Puppet and System Performance: Enterprise and the Cloud
These are the kinds of books I'm talking about. I'm in for a shopping spree in December and I'm looking for books I could use to learn system administration along the lines of Apache, NginX, DNS, Bash, Puppet, Nagios, Cacti, Virtualization, Clustering, MySQL, Performance tuning, Automation, Kernel...

Second question I have for you is: If I want to be a Linux system administration in the future, what type of job should I start with so that I could progress gradually until I reach my goal? I don't have any official IT work experience worth mentioning other than fixing PCs (and this hobby gig) and while I wanted to start with CompTIA Trio in order to find an entry level job I decided it was bull and went with Cisco and Redhat. I finally quit my job and now I'm available to study and I'm starting to doubt this certification path is right for me. I know CCNA will land you a job, but is it smart to go the networking path if I want to be a Linux system administrator? On the other hand, I can't find any entry-level or junior Linux jobs out there so what good does it do to have a Redhat certificate if you ain't got any experience to back it up. Any words of wisdom?

The Technomancer

@linuxlover:

As far as books go, the gold standard for Linux applications and services are the O'Reilly books on each topic. BIND will be the DNS service you encounter most often. For performance tuning, virtualization, clustering, and automation, there's not really any books I can recommend as those are purpose-based tasks and not really catch-all ones that have good book coverage. The StackExchange sites are going to be your best resource there.

If you want to do Linux work, find Linux gigs -- networking and systems administration really are their own fields in the Linux world. There's not a ton of crossover. The entry level Linux jobs you'll want to be on the lookout for are those that have you doing technical support for a hosting company (or other company) that utilizes Linux in their environment. Many junior level sysadmins get promoted from the tech support ranks in the Linux world, and those jobs generally don't get posted often -- the senior ones that can't always be filled by internal candidates are the ones that get posted regularly.

Don't be afraid to list the hobby gig as experience on your resume-- it is, especially if you're hosting sites other than your own.

TheNewITGuy

so.. get a job you like or salary? :P I did a CoL between here and there and we're not that far a part in salary like 10-15K - Ive been debating pursuing what I enjoy or the money.. whats your take?

The Technomancer

@TheNewITGuy:

Salary until it's enough to meet your desired level of creature comforts, then do what you love so you don't hate every minute you're earning that salary. I'm not exactly super fond of the work I do right now (ad/data management companies are high-pace, high-stress), but the salary is good and I'm hoping the equity turns into enough to launch my own business off of.

Also, COL comparisons tend to break down after about $100k/yr in earnings out here. Once you make up the difference in rent and the state taxes, the other costs either even out (power is more expensive, but you use less of it thanks to the mild climate, gas is more expensive but you drive less and mass transit kicks ass, etc). Groceries are a little more expensive, etc. Below $100k/yr out here or so, the differences are enough to impact your lifestyle.

I was billing 45-50 hours a week at the contract gig before Apple, so $100k/yr out here, according to a COL calulator when moving from Dallas to the South Bay, would have nominally been a pay cut. However, I found that I had relatively the same spending power out here. Rent went up $1500/mo, but the power bill went down from over $300 in the summer in Texas to peaking at $100ish here because I'm not running the AC full bore for 8-9 months out of the year. It was a shorter commute, so the car tax + gas price increase evened out with the money saved in not driving as far, etc.

The real money out here is in stock or equity compensation, though. Vesting a quarter of a 140 share grant from Apple as RSUs then selling it at around $500 a share tacked on quite a bit to my overall earnings for that year-ish I was there. Startups are riskier, but if you find one with a good idea and good talent, they'll make you quite a bit if they sell or IPO.

Getting back to the original question though, one thing to consider is if you're doing what you love of if you're doing it because it pays well and you're good at it. If you're doing this work because you can't see yourself doing anything else, favor the type of work over salary compensation, because otherwise you'll burn yourself out badly.

If you're doing this work because it pays well and you like the lifestyle it gives you, chase the cash so you can take nice vacations and retire early.

W Stewart

Thanks for the advice. I'd give your rep again but techexams says I have to spread it around lol.