Word Entropy

[MarcoP]

Hi Koen,
these graphs are interesting thank you for sharing them! I think that using a different color, or a different shape, for VMS samples might make the graphs more readable.

Again, I believe that h2 here is conditional entropy (not second order, i.e. Bi-Word, entropy).

ConditionalEntropy=BiWordEntropy-WordEntropy
I think your h1-h2 is WordEntropy-ConditionalEntropy = WE-(BWE-WE) = 2*WE-BWE

This could explain why the graph for h1-h2 is so similar to the  graph for h1 (WE). 
   

I hope that Rene or Anton will provide a better explanation, but I guess that the striking correlation of h1-h2 with m500 is due to the fact that it removes from word entropy the component due to unpredictable word combinations (BWE): what is left is almost identical to the variability of the lexicon (TTR). The left and right outliers in the h1-h2 graphs (mostly English texts, it seems) could then be samples with exceptionally strong or weak two-word sequences repetition. 
With the exception of labels, Bi-Word repetition appears to be normal in the VMS. What is not normal is that repeating Bi-Words so often are reduplicating words ("daiin daiin" etc), but I think this is transparent to word entropy.

I also wonder if you could compare with your corpus You are not allowed to view links. Register or Login to view.: we know their code is reasonably accurate in modelling reduplication, but I would be curious to see how it works in terms of recurring Bi-Words in general.