Experiments with language corpora

[ReneZ]

Hello DONJCH,

what you write is not wrong, but it not quite as simple as that.

The entropy measures that have been presented are usually quantities computed for one specific piece of text expressed in one specific character set. In such a case, there is one entropy measure (i.e one for H1, one for H2, one for H2-H1) and there are no error bars.
These are the entropy measures for _that text_.

This changes as soon as one starts to interpret it as an entropy measure for "the language" in which the text is expressed. In that case, one could (or should) take many different sample texts in that language and establish the error bars as you suggest. One can compute sigmas, even though the distribution will not be Gaussian.

This is where it gets a bit complicated, because the entropy values depend (to some extent) on the length of the text sample, and to lesser extents on the subject matter, the author and the age of the text.

Shorter texts will have larger error bars, but there is also a deterministic part of the dependence on the length.
This is negligible for H1, is visible for H2, significant for H3, and causes prohibitive problems in computing H4 and higher in a reliable manner.

The plot that Marco included from my web site shows two effects related to this uncertainty. On the one hand, two Latin texts that were written 16 centuries apart, but of similar length (because I truncated both) show very similar values for H1 and H2 - H1.
On the other hand, different transcription alphabets for Voynichese show (partly) different statistics, because the information is 'packetised' in different manners. Still, only Eva shows a clearly significant difference.

For the many source texts used by Marco, there is exactly one sample text in each language, which is also a bit short, so we remain uncertain about size of the error bars for the respective languages.