Word Entropy

[ReneZ]

There will always be confusion between conditional and unconditional entropy. I prefer to write upper hase H for unconditional and lower case h (which is a bit of a misnomer) for conditional entropy. Of course, there is no conditional first order entropy so one can write h1 just as well as H1.

What I intended to show in my earlier post is that, all other things being equal, both the word-H2 and the word-h2 depend directly on the text length. To be safe, one has to use the same length for all texts. Clearly, the experiment by nablator above is a valid one since both texts have the same length.
Also, arbitrarily mixing up the words should bring one close to the case I was describing, namely that all word pairs are different ones.

Using word-H1 minus word-h2 instead of word-h2 brings an advantage, because this quantity should be less dependent on text length. Using my earlier model, the value would vary from 5.6 to 6.2 for a text length varying from 5000 to 50,000 words.