How We've Scaled Dropbox

What Hacker News said

Strong rec fermienrico · 2018-09 · on “How we rolled out one of the largest Python 3 migrations”

I found the Dropbox lecture [1] at Stanford one of the most riveting things ever. There is just so much technology behind Dropbox, it is staggering. There is a reason why it is so much better than iCloud sync, Google Drive, Box or OneDrive. [1] [this video]

Recommends rawnlq · 2014-10 · on “Facebook's software architecture”

This is a talk about the evolution of Dropbox's architecture from 2012: [this video] . It is incredibly detailed (down to the exact sql schema they use for file metadata etc).

Recommends bradgnar · 2017-07 · on “Ask HN: Best-architected open-source business applications worth studying?”

Applications dont really need to be well architected until they are hitting scale. Then the parts of their system that need to relieve pressure will need to be re-architected. This is almost like a case study and there are a lot of good talks on youtube from places like dropbox and facebook that explain the problem and solution. Example: [this video] If you dont want to do youtube case studies there are also books to read about distributed systems. Also reading about cloud architecture can help.

Recommends jesseendahl · 2018-03 · on “Congrats Dropbox”

Dropbox is a lot more than a GUI on top of rsync. Even purely from an engineering standpoint (ignoring product & design) that's incorrect. You might enjoy this talk: [this video]

Recommends testb · 2018-06 · on “Ask HN: Most effective path to become a solid systems practitioner?”

imo the best way is to look at what other's have built. Here's some of my favorite talks that go from 0 users to millions. Dropbox - [this video] Instagram - [LINK] Slack (bonus. not as applicable but good reminder of why initial architecture does matter) - [LINK]

Cites LrnByTeach · 2017-07 · on “Ask HN: Best-architected open-source business applications worth studying?”

> Applications don't really need to be well architected until they are hitting scale. very True, 'a system Well architected' before hitting scale is considered OVER Engineering > Then the parts of their system that need to relieve pressure will need to be re-architected. > This is almost like a case study and there are a lot of good talks on youtube from places like dropbox and facebook that explain the problem and solution. Example: [this video]

Cites srcreigh · 2021-08 · on “How database indexing works internally”

An important detail which this guide leaves out: The "actual location of the row" is usually the leaf node of another B-Tree. That is the primary B-Tree and it's indexed by the primary key. The major consequence is every non-primary index query involves dereferencing N pointers, which could mean loading N pages from disk/SSD to get leaf nodes spread out across the primary tree. Whereas if you query a range in the primary B-Tree directly, the rows are consecutive so you would only load N/M consecutive pages based on the ratio of rowsize to pagesize. That's why some people use composite keys for primary key, to get better data locality in the primary B-Tree index. See "How We've Scaled Dropbox

Cites srcreigh · 2023-05 · on “Ask HN: It's 2023, how do you choose between MySQL and Postgres?”

Postgres is >50x slower for range queries(example below) and is akin to using array-of-pointers (ie Java) whereas MySQL supports array-of-struct (C). Illustration from Dropbox scaling talk below. Sneak peek photo [1] (from [2]). Just imagine its literally 500-1000x more convoluted per B-tree leaf node. That's every Postgres table unless you CLUSTER periodically. [1]: [LINK] [2]: [LINK] Mind boggling how many people aren't aware of primary indexes in MySQL that is not supported at all in Postgres. For certain data layouts, Postgres pays either 2x storage (covering index containing every single column), >50x worse performance by effectively N+1 bombing the disk for range queries, or blocking y