It's Official - Hadoop is my Road

the_Grinch

Let me say working with Hadoop has honestly been a nightmare. Nothing ever works right the first time (as is evident in the fact that I have deployed a cluster twice). Documentation is hard to come by and what's out there is typically written for a test environment with a single node. It exposes every flaw in your configuration of servers and services leading to an almost constant banging of your head on the table you are working on. Finally, with each failure you truly begin to question your life choices and think perhaps you could just be the best ditch digger instead of working with this technology.

But alas, each victory negates all the heart ache she has caused you. Better yet, as you enter those uncharted waters you are able to help others weary seaman make their way. This is why I think my career is going down the path of Hadoop. Learning the in's and out's have been a great ride! Plus given the small size of my team I am able to not only work on the administration side, but also the operational/analysis side as well. The field seems to be wide open and it will allow me to continue in the programming, system administration, and information assurance arena while also maybe becoming a leader in these realms as it relates to Hadoop.

N2IT

I like that you are able to work on the analysis side as well.

the_Grinch

Yup, especially since that is where the market is at (or at least for perm jobs at companies). It seems most companies, wrongly in opinion, leave it up to their system administrators to setup and operate the Hadoop cluster. Now a normal system administrator can become a Hadoop administrator (it's difficult to not have the sysadmin background and become a Hadoop administrator), but to not make it a dedicated administrator position is foolish. Many things have to be fined tuned and continuously watched/troubleshot. As an example, I deployed patches and it cause two components on one node to stop. For me it took all of about 10 minutes to get it back up and running, but for a normal system admin it probably would have taken an hour or so. Now the cluster is built to handle that, but it is definitely going to cost you to be down one node (especially if perhaps that particular service was only on that one node).

Analysts need to know how to create jobs (be it map/reduce, Pig, or Hive), know how to schedule them, and how to visualize/report their findings. Given the fact that my unit is made up of 6 people (three of which will work on Hadoop) and that we lost 25% of our people without getting new ones someone has to pick up the slack. I do have some amazing plans and in two weeks we'll be doing our first data load/analysis. Good times ahead!

paul78

Interesting commentary - I look forward to hearing more about your "fun" with Hadoop.

Good luck on your new journey.

the_Grinch

Thanks! And I don't doubt a lot more "fun" will be coming!

the_Grinch

So here's the latest fun thing to occur...LDAP. A very nice engineer made a tutorial to deploy LDAP via Ambari onto one of the servers in the cluster. Little did I know that he automated a lot of the configurations and thus used a domain that I did not want to use. I figured I could change it within the config in Ambari and that would fix the issue. Nope! And there isn't a method to auto-remove it so I'm waiting to hear I could go about doing it manually. Another fun day in the world of Hadoop.

N2IT

@ Grinch

Interesting take that is how it is set up here. We have developers who manage the database while doing some front end dev projects, but the BA's including myself data model, design triggers, SPs, and other server side SQL functions, we also script utilizing C#, VB etc depending on the solution. If it's a full blown enterprise change the professional devs get that work, but proof of concepts etc all come through us first.

I really think having both sets of skills is important and you are correct, at least from my opinion, since transitioning over to the business the pay is not as big but the perks and autonomy etc are much greater. And finding a full time job is not difficult (without having to contract out).