Word Entropy

[Anton]

Again, I believe that h2 here is conditional entropy (not second order, i.e. Bi-Word, entropy).

There is much confusing jargon with entropies, because the notion of "order" can be applied to different things. The best way (in my opinion), is to use the terms "unconditional" (this can be omitted for brevity) - this is our h1, "conditional" - these are h2, h3 etc., and "joint" - these are n-grams.

Then, one can apply the notion of order to conditional and joint entropies. Our h2 would be called "first order conditional entropy", since it's conditioned by one character, h3 - "second order conditional entropy", since it's conditioned by two characters, etc. Likewise, for joint entropies. Bigram would be second-order, trigram - third order.

This is the scheme that many authors adopt.

In contrast to that, when we speak about "order in general", this introduces confusion about what's the talk about, and then to avoid that, "n-th order entropy" should be considered as unconditional symbol entopy when order = 1, and as conditional symbol entropies when order >=2, while joint entropies are just out of scope of consideration. When you need to invoke them, you just say "joint".

I hope that Rene or Anton will provide a better explanation, but I guess that the striking correlation of h1-h2 with m500 is due to the fact that it removes from word entropy the component due to unpredictable word combinations (BWE): what is left is almost identical to the variability of the lexicon (TTR). The left and right outliers in the h1-h2 graphs (mostly English texts, it seems) could then be samples with exceptionally strong or weak two-word sequences repetition.

Unfortunately, I did not follow all that TTR discussion, and hence can say nothing.