An Essay on Entropy: what is it, and why is it so important?

[tavie]

Hi, Koen,

Not going to lie, the entropy stuff has always made my brain hurt, so this is really helpful!  Especially as somehow before I had ended up with the completely erroneous impression that either h0 or h1 was about word pair predictability and h2 letter pair predictability.  

Would you be able to answer two questions through the same simple format?

1. To what extent and why is it important when assessing h2 to divide it by h1?  I think this is what I saw in the other works on entropy.   Why does the score that reflects how much your tea drinking follows certain patterns need to be divided by the score that reflects how even/varied your choice of tea tends to be?  
    
   
2. Why does the entropy problem rule out abbreviation in the manuscript?  Or at least rule it out as a key mechanism?  I know I read it said somewhere that abbreviation can't be part of the solution due to the entropy problem.  It might have been Rene's site but it was a couple of years ago, and I can't exactly recall.  I don't think it was saying that there's absolutely no abbreviation, rather that abbreviation can't be common in the text because otherwise the entropy score would become even worse.  Would "unfolding" abbreviations out (were we ever able to do it) really make character pairs more predictable?  Surely it might in theory reveal a greater diversity of character pairs?  
   

I suspect there is a really intuitive/common sense answer to both questions that I'm not seeing; grateful if you can help!