Experiments with language corpora

[MarcoP]

Still it's interesting to see that Korean script produces such an outlier value. I still don't understand why

We can try expanding on Rene's comment about Korean:

Total characters: 3344
Nr of different characters: 303
H1 = 7.09
H2 = 9.01
Difference: 1.92 (conditional entropy)
...
What is happening here is that a very large number of syllables occur exactly once in this relatively short text.
That means that each of these is followed by exactly one other syllable, meaning that this second syllable (in this text) is completely predictable. The average information carried by these second characters is low, and this is precisely the meaning of the conditional entropy.

Effectively, character pairs are under-sampled, or in other words, the text is too short to give a good measure of the conditional entropy.

Conditional entropy measures how difficult it is to "guess" the next symbol on the basis of the immediately preceding one. Typically, in a language encoded with the Latin alphabet, you can tell that the symbol after a vowel will likely be a consonant and vice versa. Things become slightly more predictable with languages that use the Latin alphabet to represent a larger set of sounds: this requires the use of digraphs to represent sounds that do not belong to Latin. For instance, in German "ch" is very common: when you see a 'c' you can predict that it will be followed by 'h' and you will be right 95% of the times (and most of the remaining 5% will be 'ck').

To be able to measure conditional entropy, you need a sample that includes a reasonably complete set of digraphs. In the case of an alphabet with 26 different symbols, the possible digraphs are less than 26*26=676 (a number of these -like 'yy' or 'fz' in English- will be absent or so rare to be substantially irrelevant). The typical UDHR file is about 10.000 characters long: enough to estimate digraph distribution on the Latin alphabet.

Since the Korean sample has more than 300 different syllabic symbols, the possible combinations of two symbols are close to 100.000. The syllable-encoded text is about 3000 symbols long (obviously much shorter than a text represented in the Latin alphabet, where each symbol roughly corresponds to a single sound). The length of the sample is about 1/30 of the number of possible digraphs: as Rene says, a severe under-sampling. It would be like considering a 20 characters long sample of (say) German. We are not even seeing the whole alphabet: the entropy of a single character appears to be close to that of a digraph. Both H1 and H2 are under-estimated, but this is more serious for H2, which works on the largest digraph space. In such a tiny sample, both characters and digraphs will mostly occur only once or never at all. Things look very predictable: some combinations can occur (all basically with the same 1 frequency) and others cannot occur.
If you compute conditional entropy on this German fragment:
"die eltern haben ein"
you get a value below 1.  You can still get some very rough idea of H1 ('e' and 'n' are the most frequent symbols), but digraph structure is lost.