EMR Training: Running Jobs (3 of 11)
Why Stack Overflow recommends it
Stack Overflow users link it when answering or asking how to actually run a Hadoop job on EMR, using it as a walkthrough of uploading a jar and data to S3 and launching a job flow.
Recommended by 1 people across 1 different months, 2013 to 2014.
What it is
Part 3 of an 11-part Amazon Web Services training series on Elastic MapReduce, presented by Ken Krugler. It walks through creating an EMR job flow, monitoring it while it runs, and retrieving the results.
What people said
Strong rec Vidya · 2013-11 · Stack Overflow · on “How to get data from S3 and use them for Elastic map reduce/ where to write codes?”
Here is one approach. Use a distribution from Cloudera, MaprR, or wherever and use the versions (jars) of Hadoop available in the distribution. Make sure you test your jobs thoroughly locally so you feel confident everything works. This is because you will be charged at an hourly rate (per machine) by Amazon even if your job only goes for 30 seconds before failing. Once you are confident, create an "uber jar" containing all your code and all the classes in the Hadoop jars you used. Upload the jar and data to S3 as described in this excellent tutorial [this video] . EMR works seamlessly with S3. Run the job as described in the tutorial. If something goes wrong, wait a while after the job is done t
Cites Amar · 2013-02 · Stack Overflow · on “Hadoop and Amazon Web Services”
For hadoop, there is an awesome talk on Hadoop's ecosystem on AWS and EMR by the AWS team : [LINK] Then there are a series of videos tutorial they have on EMR training: [LINK] [LINK] [this video] [LINK] [LINK] [LINK] [LINK] [LINK] [LINK] [LINK] [LINK] Update : You can read this book as well : Learning Big Data with Amazon Elastic MapReduce [LINK] The Cloudera team supports deployment to AWS, and provide a whole range of documentation. Try searching for Cloudera and AWS on google, and read the documentation here [LINK] .
Cites Dhoha · 2014-03 · Stack Overflow · on “how to build a job jar for hadoop Mapreduce job in AWS”
I'm trying to run a mapreduce code example on AWS. This is the link for the code sample [LINK] However, I'm pretty new for these things. In fact, they did write in the Readme file that I should build a job jar file from the code sample. But, still didn't get how could I build a job jar. I'm following also these videos that explain how to run a job in EMR [this video] But they didn't tell also how to get this important jar file to start the work. Any help