Messaging at Scale at Instagram
Why Stack Overflow recommends it
Stack Overflow users link it when asking or answering questions about Celery task design, fanout-on-write feeds, queue reliability and worker concurrency at scale.
Recommended by 2 people across 2 different months, 2013 to 2014.
What it is
A PyCon 2015 conference talk by Rick Branson on how Instagram built messaging infrastructure as activity grew from thousands to hundreds of millions of events per day. It covers using Celery, RabbitMQ and Redis in a Python stack, including fanout-on-write for home feeds and choosing between process, thread and green-thread workers.
What people said
Recommends Philip Cristiano · 2013-04 · Stack Overflow · on “How to handle long asynchronous requests with pyramid and celery?”
RabbitMQ has flow control [LINK] built into the QoS. If RabbitMQ cannot handle the publishing rate it will adjust the TCP window size to slow down the publishers. In the event of too many messages being sent to the server it will also overflow to disk. This will allow your consumer to be a bit more naive although if you restart the connection on error and flood the connection you can cause problems. I've always decided to spend more time making sure the publishers/consumers could work with multiple queue servers instead of trying to make them more intelligent about a single queue server. The benefit is that if you are really overloading a single server you can just add another one (or anothe
Recommends deadlock · 2014-01 · Stack Overflow · on “Django, Celery, Redis, RabbitMQ: Chained Tasks for Fanout-On-Writes”
I've been watching Rick Branson's PyCon video: Messaging at Scale at Instagram [this video] . You might want to watch the video in order to answer this question. Rick Branson uses Celery, Redis and RabbitMQ. To get you up to speed, each user has a redis list for their homefeed. Each list contains media ID's of photos posted by the people they follow. Justin Bieber for example has 1.5 million followers. When he posts a photo, the ID of that photo needs to be inserted into each individual redis list for each of his followers. This is called the Fanout-On-Write approach. However, there are a few reliability problems with this approach. It can work, but for someone like Justin Bieber or Lady Gaga who
Cites Phill Pafford · 2013-05 · Stack Overflow · on “message queue - smallest amount of work for a consumer”
This question is about Architecture / Design: I have a consumer that needs to perform x numbers of tasks, when should I think about breaking the consumer up into smaller task and/or adding the smaller tasks into their own consumer? Example: Consumer FooBarShipping does adds database entry for reporting adds database entry for account adds database entry for shipping generates shipping invoice ( PDF or other formats if needed ) creates notification for shipping creates notification for account So my question is, when would I break out the bullet points into smaller consumers? The consumer as is runs fine but I feel it's getting too large and needs to be refactored into smaller, more manageabl
Cites securecurve · 2013-09 · Stack Overflow · on “Celery concurrency configuration for io/cpu bound task”
This question seems to me it needs an answer comparing multi-process/thread versus green threads; however, generally speaking in the context of celery concurrency, in doesn't make a difference using either, unless you have limited resources (and too many tasks), or you are making too many outbound connections and you are i/o-bound, then you will have to go "green" and use eventlet. A good idea, I have seen Instagram presenting in last PyCon 2013 ( Messaging at Scale at Instagram [this video] ), they use both. The main usage is the threaded tasks, yet, they use the "green" approach with those tasks that does nothing but doing outbound request to other websites, like twitter, facebook, and tumbler