AWS for Lab

Anyone using AWS for their lab? Has it worked out for you? I want to deploy a personal Hadoop clusters and figure I will need five boxes (one for DNS/NTP/homebrewed apps, one for Ambari, and three as nodes). I don't plan on pumping a ton of data into it mainly just stuff for testing. I figure it would probably run me about $1000 for all of the equipment (if I purchased physical hardware) and I don't really have the space plus the noise. I figured I could probably get all those servers on AWS for about $35 a month.

Find more posts tagged with

Comments

the_Grinch

Nevermind, my math was definitely off on this one.

--chris--

the_Grinch wrote: »

Nevermind, my math was definitely off on this one.

Well....how much was it?

impelse

I would like to know, that's something always make me thing about, I read an article you can have a server for $5 but what happen when you install more, the price goes up quickly?

kly630

There's a calculator out there. AWS actually strikes me as kind of expensive. I think there's probably ways to make it cheaper though like reserved pricing and choosing different data centers.

Amazon Web Services Simple Monthly Calculator

Edit:
5 instances of linux t1.micro 1 year no upfront money reserved compute are a little over 50 a month. Not sure how feasible it is to be locked in that long though for a test instance. On demand is about 30% more than that at $73 a month.

Claymoore

You can be very careful with your lab and keep the costs down. One trick is to make sure you have at least an hour set aside to work, since starting an instance gets billed as an hour no matter how brief it may be alive. So far I've managed to keep all of my lab work under the free tier except for paying for domain registration through Route 53. Most of my work thus far is getting familiar with the services and administration of AWS. I haven't built anything persistent yet, nor have I worked with services outside the free tier like Workspaces or Zocalo.

Are you interested in building an Hadoop cluster or just working with Hadoop? AWS Elastic Map Reduce (EMR) is the Hadoop service managed by AWS. If you are willing to work on the old hardware - and it's a lab so I assume you are - you can get the cheapest prices they offer, especially if you opt for spot instance pricing on the underlying EC2 servers instead of On-demand pricing. You may need to add in some storage on S3, but you can get a cluster running for a few pennies an hour.
AWS | Amazon Elastic MapReduce (EMR) | Pricing

NightShade03

I'm head down in a project today and tomorrow, but I can stand up a Hadoop cluster in AWS on Monday and let you know what it costs on a per hour basis (just multiple by 720 hours for your monthly costs). Work covers my AWS costs so no concerns here

the_Grinch

Excellent! Thanks all! I guess technically I don't need anything super powerful, but it would be nice.

NightShade03

So I was running a 6 node RHEL6 cluster this morning. I leveraged the "free tier" for EBS and had very little data moving in / out, but so far just my EC2 costs are $3.76 for approx 2 hours. So given this rate + checking the cost calculator my cluster would cost just north of $1000 per month. Now keep a few things in mind; no EBS costs here, no data transfer costs here, and I'm using m1.large instances for everything. You could certainly get away with small instances instead to cut costs. Hope this helps with the pricing estimates.

Update: cluster still ticking away

hdp.jpg

Mishra

Amazon's price difference comes in the value of labor and 'buy by the drink' model. To run a multi region 100-1000 VM data center, you would need a big staff. You could argue that you would need around 6-8 IT guys.

With AWS, you only need around 2-3. At 100k each for IT guys, that's a savings of 500k/year.

Claymoore

I got my bill for last month's lab expenses - $0.51. It was all for Route 53 DNS services

I have been very careful trying to keep my lab under the free tier for now. Some services will cost me money eventually. I want to try the WorkSpaces VDI offering, but I'll wait until the end of the month and try it for a prorated week. I signed up for the WorkMail preview, and that will cost me $4/month/user at some point. I don't have a need for Zocalo - oops, I mean WorkDocs - but I think I get some space for free with WorkSpaces so I'll take a look at it while testing WorkSpaces. I will run out of free tier time in 10 months or so, and I will launch some larger instances outside the free tier before then. I'm going to try and keep my costs under $50/month. Of course, I'm not trying to run a 6 node Hadoop cluster. Eventually I hope to be doing enough client work in AWS that I will not need my lab.

the_Grinch

Yeah I will definitely stick to purchasing some off lease equipment. Thanks all!