"How NOT to Measure Latency" by Gil Tene
Why Hacker News recommends it
The canonical recommendation whenever percentiles, tail latency, load testing or averages come up, described by some as life changing.
What it is
Gil Tene of Azul Systems explains common pitfalls in measuring latency and response time, including why averages mislead and the problem of coordinated omission. He introduces open source tools like HdrHistogram for better measurement and reporting.
Recommended by 9 HN commenters across 9 different months, 2015 to 2026.
What Hacker News said
Strong rec vikiomega9 · 2016-08 · on “Throughput vs. Latency and Lock-Free vs. Wait-Free”
Doesn't this depend on how "tuned" your subcomponents are? If I had a system that's supposedly heavily optimized, I would simply start seeing the worst-case latency on a higher percentile, 99.9th and so forth. Gil Tene has an excellent talk[1] on this. [1] [this video]
Strong rec Rafuino · 2019-04 · on “Averages Can Be Misleading: Try a Percentile (2014)”
This topic always leads me to think about this great talk from Gil Tene on how NOT to measure latencies (basically, don't use averages!). [this video] I'm also a huge fan of how Dormando showed latency distributions in one of his recent Memcached Extstore posts. The default is 95th percentile but you can change the percentile to what matters to you (i.e. 99th percentile if you ask me!). Scroll down to see what he did and play with it. [LINK]
Strong rec wmfiv · 2020-04 · on “Ask HN: Must Read from ACM Library?”
This a life changing video by Gil Tene on this topic. Bottom line is because of the number requests to support a single customer "request", the high percentiles are the ones that actually matter. [this video]
Strong rec leetrout · 2023-05 · on “Grasshopper – An Open Source Python Library for Load Testing”
We use locust at work but I HIGHLY recommend wrk for a very robust yet simple load testing tool. [LINK] And of course, this talk by Gil Tene is fantastic if you're interested in load testing stats [this video]
Recommends telotortium · 2017-02 · on “Unlearning descriptive statistics”
Gil Tene has an entire talk about this point called "How NOT to measure latency" ( [this video] ). In many circumstances, a web service (either server or client) has large numbers of dependencies, so a user only has to hit 95th-percentile latency for one dependency to have their overall latency significantly hurt. In other words, users may encounter the "rare" bad case on a majority of requests. Thus, even higher percentiles should be tightly controlled in order to truly keep a web service's overall latency under control.
Recommends latch · 2018-10 · on “We can do better than percentile latencies”
Worth watching is How NOT to Measure Latency: [this video] The speaker Gil Tene is also the author of the HdrHistogram which addresses this articles point: [LINK]
Recommends allanrbo · 2020-03 · on “Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers”
The "how not to measure latency" talk by Gil Tene is another really good explanation of this topic: [this video]
Recommends agallego · 2020-05 · on “Kafka Removing Zookeeper Dependency”
who cares about lower percentage tiers in the world of big data? what does the p30 or 50 even matter at large scale? See Gil Tene's talk on how not to measure latency. [this video] What matters is what most of your customers will get - p99, p999, p9999, p100.