Google I/O 2011: More 9s Please: Under The Covers of the High Replication Datastore

What people said

Cites user963263 · 2012-02 · Stack Overflow · on “How to know when updates to the Google AppEngine HRD datastore are complete?”

I have a long running job that updates 1000's of entity groups. I want to kick off a 2nd job afterwards that will have to assume all of those items have been updated. Since there are so many entity groups, I can't do it in a transaction, so i've just scheduled the 2nd job to run 15 minutes after the 1st completes using task queues. Is there a better way? Is it even safe to assume that 15 minutes gives a promise that the datastore is in sync with my previous calls? I am using high replication. In the google IO videos about HRD, they give a list of ways to deal with eventual consistency. One of them was to "accept it". Some updates (like twitter posts) don't need to be consistent with the next

Cites wtr · 2012-03 · Stack Overflow · on “What are the speed comparisons of NDB vs DB (on High Replication Datastore)?”

Taken from the Python NDB Overview [LINK] : When the application reads an entity, that entity is automatically cached; this gives fast (and inexpensive) reads for frequently-read entities. ... The NDB function that writes the data (for example, put()) returns after the cache invalidation; the Apply phase happens asynchronously. In watching on Youtube, Google I/O 2011: More 9s Please: Under The Covers of the High Replication Datastore [this video] , at 13:11-ish, the average latencies are: Master/Slave: Read: 15ms Write: 20ms High Replication: Read: 15ms Write: 45ms How significantly does NDB affect these speeds, from the app's perspective? Edit: Specifically curious about timing stats (in millise

Cites jacek2v · 2014-01 · Stack Overflow · on “Whats the meaning of the 1 per second limit on ancestor transactions in Google app engine?”

Answer: Write limit is on the guestbook (entity group). More info: Batch puts/transaction count as 1 write (limited per second) clarification: [this video]

Cites user2771609 · 2015-05 · Stack Overflow · on “What exactly is the throughput restriction on an entity group in Google App Engine's datastore?”

Ok, so you are definitely wrong about the writes per second. Here is a direct quote from a talk at Google I/O ( [this video] ). "If you use batch puts or transactions, those all count as single write, so you can get high entity per second throughput, if you fan in your writes and do them in batches". I think you have an overly simplistic view of how the datastore works. Have you read their paper on megastore, on which the datastore is built ( [LINK] )?

Cites user2771609 · 2015-06 · Stack Overflow · on “What exactly is the throughput restriction on an entity group in Google App Engine's datastore?”

I'm quite certain that you are wrong about the first point. As I said in the comments to another answer -- Here is a direct quote from a talk at Google I/O ( [this video] ). "If you use batch puts or transactions, those all count as single write, so you can get high entity per second throughput, if you fan in your writes and do them in batches".

Cites user2771609 · 2015-09 · Stack Overflow · on “Does giving a Google App Engine (Java) entity a parent guarantee consistency?”

@ZigMandel. You have misunderstood the documentation. The part you are referring to is recommending you keep the entity group small because of the write throughput limit (contention). If your write throughput is small you can store as much as you want in a single entity group. Here is a link to the Google I/O talk that says explicitly that entity groups can be of arbitrary size (at 6:16 mark): [this video]

Cites user2771609 · 2015-09 · Stack Overflow · on “Does giving a Google App Engine (Java) entity a parent guarantee consistency?”

Writing entities with the same ancestor (parent) entity DOES really guarantee consistency. How? You were wondering how this can be: "It seems that the parent entity would have the same issue as the children - That multiple instances could exist at different datastores." The answer is that there is a very sophisticated algorithm (google for Megastore papers, and the PAXOS algorithm, if you are interested in the technical details) that implements ACID transactions on entity groups even if the entities are on different machines ! Why restrict ACID to entity groups? You may be wondering then why they don't do this for the entire datastore. It seems like they figured out how to do ACID transactio