Experiments with language corpora

[ReneZ]

Based on a closer inspection, my suspicion about the Korean text turned out to be wrong.

The problem is that the conditional entropy (H2 - H1) is only a small part of the picture.

The UDHR text in Korean is indeed expressed in syllables, using UTF-8 encoding. The characters of all Hangul syllables are in the range (hex) AC00 to D7A3, and use 3 bytes in UTF-8.

In the experiments I removed all digits, replaced punctuation by spaces and collapsed consecutive spaces to single spaces.

If this text is processed on a bytes basis, the statistics  (without counting spaces) are:
Nr of different characters (=bytes): 63
H1 = 5.03
H2 = 7.84
Difference: 2.81  (conditional entropy)

If it is processed on a character (syllable) basis, the statistics (again without spaces) are:
Total characters: 3344
Nr of different characters: 303
H1 = 7.09
H2 = 9.01
Difference: 1.92 (conditional entropy)

This last value is very similar to the one reported by Marco.
While this is low (and close to Voynichese), the absolute entropy values are extremely high.

What is happening here is that a very large number of syllables occur exactly once in this relatively short text.
That means that each of these is followed by exactly one other syllable, meaning that this second syllable (in this text) is completely predictable. The average information carried by these second characters is low, and this is precisely the meaning of the conditional entropy.

Effectively, character pairs are under-sampled, or in other words, the text is too short to give a good measure of the conditional entropy.