Word Entropy

[ReneZ]

Thanks, that is very helpful.

I think that something has gone wrong with the creation of this file, but only in some parts.

The word-h0 is a direct reflection of the number of word types in the text, namely the base-2 logirithm of this number. For some texts, the lines seem to have been swapped.

One such group is lines 67 to 76. Another one is lines 181 to 198.

Another problem is that there are many texts that have quite an anomalous value for the ratio of word types over word tokens. In some cases these still have a normal m500.

I attach a modified version of the Excel file where I added a few columns to the page 'WordEntropy'.

I compute 2 ** h0 which should be equal to the number of word types.

I also compute 2 ** (h1+h2) which should be close to, but less than the number of word tokens.

You can see the problem areas.


  Koen MATTR share file_RZ.xlsx (Size: 676.26 KB / Downloads: 29)