Zach Tellman - Everything Will Flow

What Hacker News said

Strong rec stingraycharles · 2019-02 · on “Fearless Concurrency: Clojure, Rust, Pony, Erlang and Dart”

Yeah this is true, typically you want to keep blocking functions outside this. By default, core.async allocates as many threads as cores in your CPU. In addition to this, I personally find core.async to deal poorly with flow control (e.g. slow subscribers slowing down the entire flow for every subscriber), and seems to not have a lot of telemetry / it’s difficult to find out what’s going on in the thread pool behind it. I’ve personally settled on Zach Tellman’s Manifold library, which provides a much more sane abstraction and is fully compatible with core.async. For some addition thought and material on these topics, I highly recommend watching this talk by Zach, “Everything Will Flow” [VIDE

Recommends cabacon · 2015-06 · on “What happens when you add a new teller? (2008)”

See also a very nice video from ClojureWest about queues in system architectures: [this video] The queue are everywhere - your messaging queue, the threadpool, the hardware threads, and other layers of the stack and APIs you use. The video adds the interesting detail that as you add more tellers (workers) you learn of impending disaster only in the outlier p99 (or higher) latencies; by the time your p85 latency rises, you're already about to stall out.

Recommends lkrubner · 2016-03 · on “Ask HN: What are some examples of beautiful software?”

Anything by Zach Tellman: [LINK] [LINK] In terms of his ideas, watch "Always be composing": [LINK] And he offers some great thoughts about queues and backpressure in "Everything will flow": [this video]

Recommends susan_hall · 2016-08 · on “Notes on Distributed Systems for Young Bloods (2013)”

On dealing with backpressure, I would recommend what Zach Tellman writes about in his video "Everything will flow": [this video] Tellman also tells his own adventure about a system something like this: "a system that would latch up into an overload state because in overload it would log errors at a severity high enough to force the log file to be flushed on every line" He talks about some problems he faced when writing to Amazon S3, when S3 stopped accepting requests. It's a fascinating talk.

Recommends lkrubner · 2017-08 · on “Linux Load Averages: Solving the Mystery”

No. Check out this video by Zach Tellman. He talks about queues and how they break down under load. One of the least intuitive things he points out is that when you have more processors, the breakdown tends to be more of a step function: everything is running smoothly till the moment that it isn't. [this video] The point he makes arises from basic queue theory and is applicable to all kinds of systems, and how those systems react to load. It's got little to due with particular hardware and everything to do with basic math.

Cites lkrubner · 2018-11 · on “Too small to Kafka but too big to wait: Really simple streaming in Clojure”

I like this essay but I think it is very strange that he doesn't link to the talk that Zach Tellman gave in which Tellman talks about the origins of Factual's Durable Queues: [this video] Factual's Durable Queues are a fairly minimal queue effort that is useful in cases where you need some minimal way of handling back pressure, because a service downstream has failed. In his talk, Tellman talks about the fact that he needed to write a lot of data to AWS S3, and sometimes S3 stops accepting writes, so he needed an easy re-try mechanism. I used Durable Queues everywhere in my code and find it very handy. It's useful when you need a very minimal queue that still has the backing of a file system.

Cites lkrubner · 2022-06 · on “Complex systems collapse faster”

Queue theory suggests the more nodes you add to handle your queue then the longer your queue can survive (facing rising traffic) but the faster it will collapse when it finally runs out of capacity. Zach Tellman went over this in one of his talks and he had a nice graphic to demonstrate what this looks like: [this video] I suppose any complex system has dynamics like that, more capacity to absorb load, but then a faster collapse when capacity is surpassed. Hemmingway's Laws Of Motion applies: ”How did you go bankrupt?” Bill asked. ”Two ways,” Mike said. ”Gradually and then suddenly.” Put a bit differently by the economist Rudiger Dornbusch: ”The crisis takes a much longer time coming than you thi