Google I/O 2010 - Data pipelines with Google App Engine

What people said

Recommends spidee · 2010-07 · Stack Overflow · on “Google App engine(python) Updating a db.StringListProperty contention/concurrency issues”

I have now found the solution for this using a fork-join-queue. Their is a post on google IO 2010 - regarding how this is done: link text [this video]

Recommends Calvin · 2011-03 · Stack Overflow · on “Task queue filled”

Sounds like you've got a "fork bomb" in your task queue code. If you add tasks within a task you can accidentally create a huge number of tasks if an exception is being raised after the task is added. The original task gets retried, the new task gets added again (and possibly does its own bomb), the exception gets raised again, the task gets retried, etc. One way to prevent this is to give the new task a specific name so that it can't be added to the queue twice. Brett Slatkin talks about this in one of his Google I/O lectures ( here [this video] at 7:58).

Recommends tesdal · 2013-01 · Stack Overflow · on “Best practice to release memory after url fetch on appengine (python)”

Use the task queue for urlfetch as well, fan out and avoid exhausting memory, register named tasks and provide the event_list cursor to next task. You might want to fetch+process in such a scenario instead of registering new task for every process, especially if process also includes datastore writes. I also find ndb to make these async solutions more elegant. Check out Brett Slatkins talk on scalable apps [LINK] and perhaps pipelines [this video] .

Recommends Nicholas Franceschina · 2015-05 · Stack Overflow · on “How to execute tasks in (FIFO) order using Google App Engine Task Queues?”

Consider using the Pipelines API: [LINK] They have already done the work for you: [LINK] class LogWaitLogInOrder(pipeline.Pipeline): def run(self, message1, message2, delay): with pipeline.InOrder(): yield LogMessage(message1) yield Delay(seconds=delay) yield LogMessage(message2) yield LogMessage('This would happen immediately on run') The pipelines api also gives you reporting/feedback/notification/etc. But if that's overkill for your particular needs, then just do as @Paul suggested, just create the next task when the first one completes and hope for the best video reference: Google I/O 2010 - Data pipelines with Google App Engine: [this video]

Recommends Nicholas Franceschina · 2018-08 · Stack Overflow · on “Does Google App Engine support multiprocessing via python, and does the DB support multiple writes in localhost?”

Regarding database lock: Updates are actually done within a datastore transaction and NDB by default will retry the operation three times before failing altogether. It is recommended you only update an entity group once per second at the most. If you are seeing database locks, then you're probably doing something wrong. We implemented a version of the "fork-join queue" described by Brett Slatkin back in 2010 data pipelines talk, which is a method of "joining" many updates to the same entity such that they can all be applied at once at a controlled rate: [this video] also, see the discussion going on here: How to deal with eventual consistency in fork-join-queue [LINK]

Cites Jason Hall · 2010-06 · Stack Overflow · on “Database design - google app engine”

As you've noticed, this design doesn't scale. It requires 4 (!!!) DB queries to render the page. That's 3 too many :) The prevailing notion of working with the App Engine Datastore is that you want to do as much work as you possibly can when something is written, so that almost nothing needs to be done when something is retrieved and rendered. You presumably write the data very few times, compared to how many times it's rendered. Normalization is similarly something that you seem to be striving for. The Datastore doesn't place any value in normalization -- it may mean less data incongruity, but it also means reading data is muuuuuch slower (4 reads?!!). Since your data is read much more ofte

Cites Jaroslav Záruba · 2010-06 · Stack Overflow · on “Why does the task name contain "now / 30"?”

In the video/PDF from "Data pipelines with Google App Engine" Brett puts "now / 30" into the task name noting that he will explain the reason later, but somehow he never does. :) [this video] task_name = '%s-%d-%d' % (sum_name, int(now / 30), index) Do you have any idea about the reason? Does it have anything to do with the 7 day period in which one can't re-use task names? Link to the session page [LINK]

Cites v_y · 2010-06 · Stack Overflow · on “app engine data pipelines talk - for fan-in materialized view, why are work indexes necessary?”

I'm trying to understand the data pipelines talk presented at google i/o: [this video] I don't see why fan-in work indexes are necessary if i'm just going to batch through input-sequence markers. Can't the optimistically-enqueued task grab all unapplied markers, churn through as many of them as possible (repeatedly fetching a batch of say 10, then transactionally update the materialized view entity), and re-enqueue itself if the task times out before working through all markers? Does the work indexes have something to do with the efficiency querying for all unapplied markers? i.e., it's better to query for "markers with work_index = " than for "markers with applied = False"? If so, why is that? F

Cites Jaroslav Záruba · 2010-06 · Stack Overflow · on “Why is only the suffix of work_index hashed?”

I'm reading through the PDF that Brett Slatkin has published for Google I/O 2010: " Data pipelines with Google App Engine [LINK] ". In this video [this video] (the Fan-in part) Brett says that the work_index has to be a hash, so that 'you distribute the load across the BigTable'... and this is how work_index is created: work_index = '%s-%d' % (sum_name, knuth_hash(index)) ...which I guess creates something like 'mySum-54657651321987' I do understand the basic idea, but is why only one half of work_index is hashed? Is it important to hash only part of it leaving the suffix out? Would it be wrong to do md5('%s-%d' % (sum_name, index)) so that the hash would be like '6gw8....hq6' ? I'm Java guy so I

Cites Rob Curtis · 2012-09 · Stack Overflow · on “What happens when an async put, results in a contention exception, after the request has ended, on Appengine with NDB?”

As far as I understand, using @ndb.toplevel causes the handler wait for all async operations to finish before exiting. From the docs: As a convenience, you can decorate the request handler with @ndb.toplevel. This tells the handler not to exit until its asynchronous requests have finished. This in turn lets you send off the request and not worry about the result. [LINK] So by adding @ndb.toplevel that the response doesn't actually get returned until after the async methods have finished executing. Using @ndb.toplevel removes the need to call get_result on all the async calls that were fired off (for convenience). So based on this, the request would still return 500 if the async queries faile