A Short Introduction to Entropy, Cross-Entropy and KL-Divergence
Why Mathematics Stack Exchange recommends it
Mathematics Stack Exchange users link it when explaining or asking about the intuition behind cross-entropy and relative entropy, especially the link between code length and predicted probability.
Recommended by 2 people across 2 different months, 2019 to 2023.
What it is
A short explainer by Aurélien Géron that builds up entropy, cross-entropy and KL-divergence from Shannon's information theory, using a weather-message coding example. It connects the ideas to why cross-entropy is used as a loss function when training machine learning classifiers.
What people said
Recommends Zaid Khan · 2020-01 · Mathematics Stack Exchange · on “How can I think of (visualize) 0.5 bits of information?”
@JyrkiLahtonen I was actually learning it for understanding how Cross-Entropy is used as a cost function in deep learning. Basically the whole idea of deciding the difference between a true probability distribution and using actual distribution by finding the difference in the bits used vs. bits which actually were needed to carry information. You can watch it here [this video]
Recommends Enes · 2020-10 · Mathematics Stack Exchange · on “What is the reasoning behind the definition of relative entropy?”
As you can see in this [this video] video. Cross entropy: H(p,q) can be interpreted as average message length that is sent to receiver based on predicted q distribution. In information theory, to compress given massage in most effective way, we look at its probability distribution. So basically, entropy: H(p) of the given probability distribution is our ultimate limit. For example, if I was sending 4 bits of data on average to receiver about the wheater of Istanbul, and also if I know the entropy of my message words is actually 3.2, it means that I can actually can compress my message more. Thus, the difference between average message lenght that was sent and actual entropy of the message is equa
Cites Mr.Robot · 2019-10 · Mathematics Stack Exchange · on “Relationship between code length and predicted distribution”
Problem In a video I just watched (as could be seen here [this video] , between 7:10 - 7:30), the author claims that the number of digits one uses to encode a particular weather condition reflects his/her prediction about the weather distribution. For example, if two digits are used, then he/she believes the probability of that weather condition is $\frac{1}{2^2}=0.25$ . Similarly, if five digits are used, then the probability should be $\frac{1}{2^5}=0.03125$ . As much as I could understand this from high level, i.e. if something happens quite often, then using fewer digits will surely save your transmission cost. However, I do not understand the mathematical reasoning of this correspondence bet
Cites Yiffany · 2023-08 · Mathematics Stack Exchange · on “Understand cross entropy's calculation”
Screenshot from this video. [this video] Why he calculate predicted probability by using $\frac{1}{\text{bit length}^2}$ , what does those calculated probability means? I added up all the predicted distribution and it is $93.x\%$ not $100\%$ but true distribution is $100\%$ why not match? what is the difference means? Is this difference KL-divergence? [LINK]