The 'Chinese' Theory: For and Against

[Jorge_Stolfi]

Stolfi, have you actually measured the entropy of monosyllabic languages? How do they need to be written down to achieve a conditional character entropy that's well under 3, while at the same time working with a very limited alphabet?

Rene and others have re-posted the analyses.  As I can see, indeded the character entropies of Mandarin in Pinyin with numeric tones and Vietnamese in VIQR come close to those of Voynichese.

But let me insist that the character entropy is not an interesting measure, because it is highly dependent on the encoding of sounds into characters, and in particular on what one considers a character.   The h_k entropies of German would be higher if one used "x", "k", "ö" etc instead of "sch", "ck", "oe", etc.  Conversely the h_k entropies of Vietnamese and Mandarin would be higher if one encoded the diacritics with the main vowel as a single character, instead of 2-3 characters like VIQR and Pinyin do.  In the VMS, are qo, ee, iin, cTh single characters, or clusters of separate characters?  The choice will affect the h_k entropies.

The word entropy is a more useful measure, because it does not depend on the encoding or encryption, as long as it is one-to-one (such as a simple substitution cypher, Pig Latin, or writing words bacwards).  

An encoding that is many-words-to-one-word (such as writing Mandarin or Vietnamese without tone information) would decrease the word entropy.  An encoding that is one-word-to-many-words (such as a Vigenère cipher) would increase the word entropy, and would also mess up the Zipf plots.  Since the word entropy and Zipf plots of Voynichese are quite "natural", it is still quite possible that it is a natural language with an encoding that is somewhat verbose but still mostly one-word-to-one-word.

By the way, the h_k entropies (uncertanty of next character given the previous k characters) are not the best ones.  More useful would be e_n = the entropy at position [n] in the word, given all the previous n characters of the word.  The encoding will affect this metric too, but since the sum of all e_n is the (zero-context) word entropy, it can only affect how this total is spread along the positions n.