Parsing JSON Really Quickly: Lessons Learned
Why Hacker News recommends it
HN points to it as the best explanation of simdjson internals and of why fast parsing is rarely disk-bound.
What it is
Daniel Lemire's QCon talk on how simdjson parses gigabytes of JSON per second. It covers SIMD vectorization, branch prediction and the lessons learned building a fast parser.
Recommended by 3 HN commenters across 3 different months, 2020 to 2026.
What Hacker News said
Recommends dehesa · 2020-05 · on “So you want to write your own CSV code (2014)”
I can relate. I have been one of those who started writing a CSV library and went down the rabbit hole of edge cases and configurability. Looking back (and only thinking on productivity) my time would have been better spent doing something else. I did, however, learn quite a lot along the way: - how text is represented in a computer (and all its caveats), - how to identify and tune slow code, - how to vectorize code (thanks to Daniel Lemire's [this video] ), - how to encapsulate code better, - how to use some great CSV tools out there (thanks to Leon's [LINK] ) - how to better manage and communicate an open source project. All in all, I am still not sure it was a mistake. Moreover when Swift didn
Recommends Rendello · 2021-10 · on “Word-Aligned Bloom Filters”
I find these optimizations fascinating. Anyone familiar with Lemire likely knows about this, but I listened to a podcast episode[1] with him a few days ago and learned about `simdjson`, the tool he authored that parses JSON at 25x the speed of the standard C++ library.[2][3] It's worth looking at if you're into this sort of thing. 1. [LINK] 2. [this video] 3. [LINK]
Recommends D4ckard · 2024-01 · on “The One Billion Row Challenge”
Daniel Lemire has an interesting talk on this: [this video] . They key takeaway is that the disk is rarely the bottleneck.
Cites avian · 2020-08 · on “Simdjson: Parsing gigabytes of JSON per second”
The GitHub page links to a video that explains some of the internals [1]. Can someone comment on the result that they show at 14:26? My understanding is that they run a code that does 2000 branches based on a pseudo-random sequence. Over around 10 runs of that code, the CPU supposedly learns to correctly predict those 2000 branches and the performance steadily increases. Do the modern branch predictors really have the capability to remember an exact sequence of past 2000 decisions on the same branch instruction? Also, why would the performance increase incrementally like that? I would imagine that it would remember the loop history on the first run and achieve maximum performance on the seco
Cites ubedan · 2026-07 · on “GigaToken: ~1000x faster Language model tokenization”
Github: [LINK] It showcases an especially ingenious scheme for escaping json strings, as well as the other tokens necessary to parse json. Daniel Lemire - One of the authors at QCon 2019: [this video] Another Youtube video that explains it: [LINK]