It's Official - Hadoop is my Road

the_Grinchthe_Grinch Member Posts: 4,165 ■■■■■■■■■■
Let me say working with Hadoop has honestly been a nightmare. Nothing ever works right the first time (as is evident in the fact that I have deployed a cluster twice). Documentation is hard to come by and what's out there is typically written for a test environment with a single node. It exposes every flaw in your configuration of servers and services leading to an almost constant banging of your head on the table you are working on. Finally, with each failure you truly begin to question your life choices and think perhaps you could just be the best ditch digger instead of working with this technology.

But alas, each victory negates all the heart ache she has caused you. Better yet, as you enter those uncharted waters you are able to help others weary seaman make their way. This is why I think my career is going down the path of Hadoop. Learning the in's and out's have been a great ride! Plus given the small size of my team I am able to not only work on the administration side, but also the operational/analysis side as well. The field seems to be wide open and it will allow me to continue in the programming, system administration, and information assurance arena while also maybe becoming a leader in these realms as it relates to Hadoop.
WIP:
PHP
Kotlin
Intro to Discrete Math
Programming Languages
Work stuff

Comments

  • N2ITN2IT Inactive Imported Users Posts: 7,483 ■■■■■■■■■■
    I like that you are able to work on the analysis side as well. ;)
  • the_Grinchthe_Grinch Member Posts: 4,165 ■■■■■■■■■■
    Yup, especially since that is where the market is at (or at least for perm jobs at companies). It seems most companies, wrongly in opinion, leave it up to their system administrators to setup and operate the Hadoop cluster. Now a normal system administrator can become a Hadoop administrator (it's difficult to not have the sysadmin background and become a Hadoop administrator), but to not make it a dedicated administrator position is foolish. Many things have to be fined tuned and continuously watched/troubleshot. As an example, I deployed patches and it cause two components on one node to stop. For me it took all of about 10 minutes to get it back up and running, but for a normal system admin it probably would have taken an hour or so. Now the cluster is built to handle that, but it is definitely going to cost you to be down one node (especially if perhaps that particular service was only on that one node).

    Analysts need to know how to create jobs (be it map/reduce, Pig, or Hive), know how to schedule them, and how to visualize/report their findings. Given the fact that my unit is made up of 6 people (three of which will work on Hadoop) and that we lost 25% of our people without getting new ones someone has to pick up the slack. I do have some amazing plans and in two weeks we'll be doing our first data load/analysis. Good times ahead!
    WIP:
    PHP
    Kotlin
    Intro to Discrete Math
    Programming Languages
    Work stuff
  • paul78paul78 Member Posts: 3,016 ■■■■■■■■■■
    Interesting commentary - I look forward to hearing more about your "fun" with Hadoop.

    Good luck on your new journey.
  • the_Grinchthe_Grinch Member Posts: 4,165 ■■■■■■■■■■
    Thanks! And I don't doubt a lot more "fun" will be coming!
    WIP:
    PHP
    Kotlin
    Intro to Discrete Math
    Programming Languages
    Work stuff
  • the_Grinchthe_Grinch Member Posts: 4,165 ■■■■■■■■■■
    So here's the latest fun thing to occur...LDAP. A very nice engineer made a tutorial to deploy LDAP via Ambari onto one of the servers in the cluster. Little did I know that he automated a lot of the configurations and thus used a domain that I did not want to use. I figured I could change it within the config in Ambari and that would fix the issue. Nope! And there isn't a method to auto-remove it so I'm waiting to hear I could go about doing it manually. Another fun day in the world of Hadoop.
    WIP:
    PHP
    Kotlin
    Intro to Discrete Math
    Programming Languages
    Work stuff
  • N2ITN2IT Inactive Imported Users Posts: 7,483 ■■■■■■■■■■
    @ Grinch

    Interesting take that is how it is set up here. We have developers who manage the database while doing some front end dev projects, but the BA's including myself data model, design triggers, SPs, and other server side SQL functions, we also script utilizing C#, VB etc depending on the solution. If it's a full blown enterprise change the professional devs get that work, but proof of concepts etc all come through us first.

    I really think having both sets of skills is important and you are correct, at least from my opinion, since transitioning over to the business the pay is not as big but the perks and autonomy etc are much greater. And finding a full time job is not difficult (without having to contract out).
Sign In or Register to comment.