Google I/O 2011: App Engine MapReduce

What people said

Recommends Daniel Roseman · 2011-08 · Stack Overflow · on “running hadoop on Google app engine?”

Sort of. You can't use the actual MapReduce framework - the architecture is too incompatible with AppEngine. However, there is an equivalent system built specficially for GAE - appengine-mapreduce [LINK] . That site is a bit confusing, as the first version of the code only supported mappers, without the subsequent reduce step - recently they released a version with full mapreduce support, but some of the documentation still referes to the earlier mapper-only one. The best introduction is the GoogleIO talk from Mike Aizatskyi [this video] .

Cites Pawel Markowski · 2011-05 · Stack Overflow · on “Simple counter example using mapreduce in Google App Engine”

I'm somewhat confused with the current state of mapreduce support in GAE. According to the docs [LINK] reduce phase isn't supported yet, but in the description of the session from I/O 2011 ( [this video] ) it's written "It is now possible to run full Map Reduce jobs on App Engine". I wonder if I can use mapreduce in this task: What I want to do: I have model Car with field color: class Car(db.Model): color = db.StringProperty() I want to run mapreduce process (from time to time, cron-defined) which can compute how many cars are in each color ans store this result in the datastore. Seems like a job well suited for mapreduce (but if I'm wrong correct me), phase "map" will yield pairs (, 1) for each

Cites Dan Sanderson · 2012-01 · Stack Overflow · on “Task Queues or Multi Threading on google app engine”

The basic structure for what you're doing is to have the cron job be responsible for dividing the work into smaller units, and executing each unit with the task queue. The payload for each task would be information that identifies the entities in the first set (such as a set of keys). Each task would perform whatever queries are necessary to join the entities in the first set with the entities in the second set, and store intermediate (or perhaps final) results. You can tweak the payload size and task queue rate until it performs the way you desire. If the results of each task need to be aggregated, you can have each task record its completion and test for whether all tasks are complete, or

Cites Shai Almog · 2012-09 · Stack Overflow · on “Are there any samples for appengine Java report generation?”

We are using AppEngine and the datastore for our application where we have a moderately large table of information containing a list with entries. I would like to summarize the list of entries in a report specifying how many times each one appears e.g. normally in SQL I would just use a select distinct for a column, then loop over every entry and just use select count(x) where value = valueOfEntry . While the count portion is easily done, the distinct problem is "a problem". The only solution I could find remotely close to this is MapReduce and most of the samples are based on Python. There is this blog entry [LINK] which is very helpful but somewhat outdated since it predated the reduce por

Cites Nicholas Franceschina · 2015-05 · Stack Overflow · on “Keeping Consistent Count in Google App Engine”

you could use the MapReduce or Pipeline API: [LINK] [LINK] allowing you to split your problem into smaller manageable parts whereby the library can handle all of the details of signaling/blocking between tasks, gathering the results, and handing them back to you when it's done Google I/O 2010 - Data pipelines with Google App Engine: [LINK] Google I/O 2011: Large-scale Data Analysis Using the App Engine Pipeline API: [LINK] Google I/O 2011: App Engine MapReduce: [this video] Google I/O 2012 - Building Data Pipelines at Google Scale: [LINK]