Breakout Session Presented by Dan Robinson of Heap at PGConf Silicon Valley 2015

What Hacker News said

Strong rec Judson · 2016-05 · on “TrailDB – An Efficient Library for Storing and Processing Event Data”

I was just watching this[0] excellent talk from one of the Heap Analytics engineers on their use of CitusDB + Postgres to do event processing. What use-case would TrailDB be the obvious, hands-down way to go (vs maybe Postgres + Citus)? [0]: [this video]

Recommends devbug · 2016-08 · on “Building a Scalable Postgres Metrics Backend Using the Citus Extension”

If you haven't already seen it, Dan from Heap Analytics has a great talk about how they use Postgres and Citus. [this video]

Recommends simiano · 2016-09 · on “When to Avoid JSONB in a PostgreSQL Schema”

Keep in mind that Heap collects a huge amount of data and has a huge dataset. I like this[1] talk, it gives you an overview of their architecture. Great post though, thank you for sharing. [1] [this video]

Recommends macobo · 2017-01 · on “How Discord Stores Billions of Messages Using Cassandra”

Regarding a), a sharded postgresql (i.e. with citus data) can easily accomodate that workflow with just a tiny bit of extra overhead. Re c) this strongly depends on the usecase, I've seen companies use a) to avoid split-brain problems and having to manage two data pipelines to great success at similar scales. You might find [this video] interesting.

Recommends macobo · 2017-06 · on “Running 10M PostgreSQL Indexes in Production”

You need a queue in front of your database irregardless of write latency. Otherwise you tie your availability to database availability as downtime (even for upgrades) is often unavoidable and network problems are common. Dan gave a pretty good talk about the high-level details of how/why postgres a couple of years back: [this video] . One reason of why pg is that SQL is really powerful for building complex queries.

Cites malisper · 2016-09 · on “When to Avoid JSONB in a PostgreSQL Schema”

As of right now, our schema is literally: user_id | event_id | time | data where data is a JSONB blob that contains every other piece of information about an event. Currently, we get a row estimate of one for pretty much every query. We've been able to work around the lack of statistics by using a very specific indexing strategy (discussed about in a talk Dan gave[0]) that gives the planner very few options in terms of planning and additionally by turning off nested loop joins. We are planning on pulling out the most common properties that we store in the data column, which will give us proper statistics on all of those fields. I am currently experimenting with what new indexing strategies w

Cites malisper · 2016-09 · on “When to Avoid JSONB in a PostgreSQL Schema”

That get's into our indexing strategy which Dan talks about in this talk[0]. Currently our tables aren't completely insert only, so a BRIN index wouldn't work for us, as one row in the wrong place can cause a huge amount of extra reads. [0] [this video]

Cites malisper · 2020-02 · on “Launch HN: PostHog (YC W20) – open-source product analytics”

I'm curious as to how you plan to scale PostHog to larger users. As the person who scaled Heap, here is my honest opinion of this. I think there is going to be a huge challenge ahead in scaling query performance. This was perpetually a challenge at Heap and was for a long time the main limitation on Heap's growth. The challenge was tough enough for Heap and PostHog is going to be at a huge disadvantage due to the lack of multi-tenancy. When you use Heap, your data is stored across Heap's entire cluster of machines. When you run a query, that query is ran simultaneously against every single machine in Heap's cluster. Even though your data may be taking up something like .1% of the total disk