The Zipf Mystery

What Hacker News said

Recommends Baeocystin · 2019-02 · on “German for Programmers”

I won't say it's easy , but it's easier than you might think. Word use follows a power law distribution in all languages ( Zipf's Law, [this video] is an excellent vsauce cover of the subject ), but the short of it is that within a relatively short amount of time spent studying you'll be able to pick out common words and build from there, even if you started from zero.

Recommends sillysaurusx · 2019-11 · on “Any number can start a factorial”

Your shock is warranted. It's actually a very deep question, called the Zipf mystery: [this video] No one really knows why it's true. It's also true across physical constants, not just artificial quantities made up by humans. The same type of distribution pops up in language, in nature, in numbers, popularity of opening chess moves, everywhere.

Recommends WhateverHappns · 2021-04 · on “Embrace the Grind”

> You need to know about 20,000 words in any language to sound fluent Are you sure about this? AFAIK the use of words in a language isn't distributed evenly, but according to a Pareto distribution. Thus, you can speak _okay_ knowing only ~500 words, I believe. It won't sound perfect but you'll speak and understand pretty decently. I once wrote a python script which read words from a .txt file full of french words with their english counterparts and then displayed the french word, waiting for you to type your guess. It gave instant feedback which I thought important for learning anything. The file contained the 500 most used french words according to large book surveys. This was in my last ye

Recommends its_ethan · 2024-07 · on “The mysterious maths at work in the Tour de France”

There's an entertaining "V-Sauce" video about Zipf's law for anyone wanting to learn a bit more about it and to see other examples of where it shows up: [this video]

Cites dredmorbius · 2018-02 · on “‘Success’ on YouTube Still Means a Life of Poverty”

Mass-publishing markets are very strongly given to power-law distributions. We hear that "anybody can be famous" (or successful, or ...), but everybody cannot be famous. Attention and publicity are the ultimate zero-sum games: no matter how large a given talent pool, there will only ever be ten top-ten spots, 100 top-100 spots. All you're doing by enlarging the pool is amplifying the competition for those spots. There's another side to the equation, though, which is the scope of the audience . Prior to modern technological methods, and I'm including here everything from relatively modern printing (say, 19th century onward), as well as phonograph, cinema, radio, television, and sound amplific

Cites sillysaurus3 · 2018-06 · on “Oldest domains in the .com, .net, and .org TLDs”

It's a deeper reason than that. Humans are hardwired to start with short symbols. Picture a group of humans who evolve a new language. That language will likely start with short sounds, and then progressively build up to longer sounds. [this video] Anyone who hasn't read A Mathematical Theory of Communication, you're missing out. It's quite readable and quite cool: [LINK]

Cites robbedpeter · 2021-11 · on “Unzipping Zipf’s Law (2017)”

Zipf distributions show up in randomly generated character sequences, Semantics are orthogonal. Zipf thought it was the principle of least effort, the use of language to minimize size of communication and maximize meaning. This heuristic causes the use of language to approach some ideal optimal communication, which subsequently shows up in statistics as Zipfy. The underlying reason is something rooted in information and complexity theory, and will depend on Kolmogorov complexity and systems modeling. To quote vsauce : "...Moreover, Zipf's law doesn't just mysteriously describe word use. It's also found in city populations, solar flare intensities, protein sequences and immune receptors, the

Cites cowtools · 2022-10 · on “The Internet needs the InterPlanetary File System”

The majority of seeding/downloading will be done with the (minority) most popular files, while only a minority of seeding/downloading is done for the majority of files. For example, go to TPB and look at the distribution of seeders/leechers. [this video]