Google I/O 2009 - ..Scalable, Complex Apps on App Engine
Why Stack Overflow recommends it
Stack Overflow users link it when asking how to model data, build feeds, or understand list properties and indexes in the App Engine datastore.
Recommended by 6 people across 7 different months, 2009 to 2016.
What it is
A 2009 Google I/O session by Brett Slatkin on designing larger, more complex applications on Google App Engine that still scale. It covers advanced datastore data structures and techniques such as list properties, merge-join queries, and index design, with worked examples like a microblogging feed and a friends graph.
What people said
Strong rec Peter Knego · 2011-02 · Stack Overflow · on “On GoogleEngine (Java), in JDO, how do I query a list of child objects based on the id of the parent?”
A few notes: List properties in Entities (like your List<Event> events ) are stored as a serialized ProtocolBuffer. The problem is: If this property is indexed, they are limited to 5000 elements. Every time you query the list, whole list needs to be deserialized. This is the answer to your question if you can selectively retrieve list elements: you can't. If you have multiple indexed list properties in entity then this can lead to Exploding Indexes [LINK] . If you want to understand internals of GAE datastore then this vido is a must: [this video] Solutions: Use a solution from Slatkin's video: make Calendar a parent (in datastore terms) of Event. Then add a parent condition to query: Query.setAn
Strong rec guigouz · 2011-02 · Stack Overflow · on “Selecting distinct entities across a large google app engine table”
Denormalization should solve your problem - having a last_journey reference property in your car, so everytime you start a journey, you'd also update the Car entity - this way you'd be able to query all cars and have their lastest journey on the resultset. It's worth noting that when you access last_journey, a new get() will be issued to the datastore, so if you're listing a lot of cars, you could build a list with all the last_journey keys and fetch then all at once passing that to db.get(). Scalable, Complex Apps on App Engine [this video] is definately a must watch (sadly the sound is terrible on this video)
Recommends Federico Builes · 2009-07 · Stack Overflow · on “Storing a directed graph in google appengine datastore”
Depending on the number of vertex/links you might want to just use lists instead of creating a bunch of new entities. Check the friends graph problems described in the second half of this video from Google IO 2009: [this video] If you think the number of vertex is high enough you can just create a Vertex model with a list that represents the connections.
Recommends Peter Knego · 2011-02 · Stack Overflow · on “Selecting distinct entities across a large google app engine table”
First of all I'd recommend using objectify [LINK] . JDO/JPA on appengine just fool people into thinking that appengine datastore is just a SQL database, which, as you realized, is far from the truth. If I understand correctly you have a Car which contains a List of CarJourneys? List properties on appengine are limited to 5000 entries and any time you access/change them they have to be serialized/deserialized in whole. So if you plan to have a lot of CarJourneys per Car than this will get slow. Also because appengine creates an index entry for every value in the collection this can lead to exploding indexes [LINK] . Instead, just create a property Car inside CarJourney that points to the Car
Recommends Calvin · 2011-03 · Stack Overflow · on “3 Models Entity Joins in AppEngine Python”
This Google I/O talk covers this subject really well. I believe he goes over how he would implement a twitter-like service in App Engine: [this video]
Recommends bdonlan · 2011-08 · Stack Overflow · on “How does appengine implement query on a list efficiently?”
The algorithm used internally ('merge-join') was described in the Google I/O 2009 tech talk Building Scalable, Complex Apps on App Engine [this video] . This functionality has also been available since the launch of GAE; the 'exploding indexes' only happen if you create a compound index of multiple StringListProperties. It's worth noting that this functionality is actually a bit more general than you may realize - any combination of multiple equality filters on any arbitrary combination of properties can be satisfied without any compound indices, provided they're all equality filters and you don't have a sort order. They don't all have to be from a StringListProperty, and can even be split across
Recommends stickfigure · 2014-05 · Stack Overflow · on “many to many relationship using objectify?”
Beyond the answers already posted I would suggest that you not include a link from User to the Message, for three reasons: Collections in GAE are hard limited to 5000 items. As soon as your user's inbox exceeds 5k items your app will start throwing exceptions. There is a performance cost to expanding the quantity of data in an entity; loading a bunch of 500k entities is slower than loading a bunch of 5k entities. Plus your usage of memcache will be less effective since you can fit fewer items in the same space. User objects tend to get loaded a lot . You can easily hit the transaction rate limit for a single entity (1/s). If 50 people send you a message at the same time, you will have massiv
Recommends stickfigure · 2016-01 · Stack Overflow · on “Create feed like api method with a cursor in Cloud Endpoints (google app engine)”
There are two general approaches to building a feed: Scatter-gather, or preindex. They each have their tradeoffs. Scatter-gather is usually the easiest to start, especially when you can do joins in the DB. And it's conveniently flexible - change the code and everyone sees the result immediately. However, scalability of this is severely limited, especially if your users typically follow large numbers of others. As you've noticed, it's a PITA in GAE because there are no joins (and an IN only lets you specify a few tens of items). Figuring out what to display to a user at display-time is expensive. Preindex (or "inbox") basically says, pick who sees a piece of content when the content is create
Cites wings · 2009-10 · Stack Overflow · on “A good data model for finding a user's favorite stories”
Original Design Here's how I originally had my Models set up: class UserData(db.Model): user = db.UserProperty() favorites = db.ListProperty(db.Key) # list of story keys # ... class Story(db.Model): title = db.StringProperty() # ... On every page that displayed a story I would query UserData for the current user: user_data = UserData.all().filter('user =' users.get_current_user()).get() story_is_favorited = (story in user_data.favorites) New Design After watching this talk: Google I/O 2009 - Scalable, Complex Apps on App Engine [this video] , I wondered if I could set things up more efficiently. class FavoriteIndex(db.Model): favorited_by = db.StringListProperty() The Story Model is the same, but
Cites Lukáš Lalinský · 2009-10 · Stack Overflow · on “AppEngine: Will the items stored in a StringListProperty always remain in the same order?”
I can't find it explicitly said in the documentation, but according to a Google I/O talk about the DataStore [this video] , StringListProperty will be sorted the same way you put it there.