There's a fundamental problem with word-h2.
If a text is made up of 26 different characters , then there are 676 different possible character pairs. The entropy is computed as a function of the 676 different probabilities, effectively by computing the frequency.
The longer the text, the more representative the frequencies become.
For word entropy, the number of different words keeps increasing. The number of possible word pairs actually grows faster than the text length, so in a way the sampling gets worse as the text length increases.
Edit:
One can do an estimate. Assume that a text consists of N+1 words. Then there are N different word pairs. Let's assume that all pairs are different. In that case each of the N pairs has a frequency of 1 / N, and the absolute word H2 would be 2log(N). (Base-2 logarithm).
For the computation of h2 one would need to know H1. For a text following the Zipf law, this has been estimated in a table provided here:
You are not allowed to view links. Register or Login to view.
The h2 can then be estimated (for this particular case) as the difference between the two columns H(max) - H(zipf), and for text lenghts of 5000 to 50,000 words it would run from 3.3 to 4.7 . (All barring simple mistakes).
If a text is made up of 26 different characters , then there are 676 different possible character pairs. The entropy is computed as a function of the 676 different probabilities, effectively by computing the frequency.
The longer the text, the more representative the frequencies become.
For word entropy, the number of different words keeps increasing. The number of possible word pairs actually grows faster than the text length, so in a way the sampling gets worse as the text length increases.
Edit:
One can do an estimate. Assume that a text consists of N+1 words. Then there are N different word pairs. Let's assume that all pairs are different. In that case each of the N pairs has a frequency of 1 / N, and the absolute word H2 would be 2log(N). (Base-2 logarithm).
For the computation of h2 one would need to know H1. For a text following the Zipf law, this has been estimated in a table provided here:
You are not allowed to view links. Register or Login to view.
The h2 can then be estimated (for this particular case) as the difference between the two columns H(max) - H(zipf), and for text lenghts of 5000 to 50,000 words it would run from 3.3 to 4.7 . (All barring simple mistakes).