Analysis of Voynich Average Cross-Entropy Loss

[Jorge_Stolfi]

I looked at the word-length frequency and character-frequency of the Voynich and created a random corpus of equal length which has the same word length and character frequency histograms, but are effectively random.

It has long been known that Voynichese words have a rather rigid structure.  Some early "word models" were published by Titman, Mike Roe, and Robert Firth.  Here is one of my own:
   
   
   
   
This rigid structure is why the per-character entropy of Voynichese is lower than that of English and other common languages, and much lower than that of your pseudo-Voynichese (a Markov generator of order zero).

But character statistics are generally not very useful (unless one makes very specific assumptions about the language and encoding). They are very hard to interpret and can easily lead people astray.  For instance, we still don't know what exactly is the the Voynichese alphabet.  Even relatively phonetic and logical spelling systems, like Italian and Spanish, have complicated rules that would make it impossible to answer that question correctly from statistics alone.  In Italian the letters "g" and "n" are common and occur independently as separate letters.  Bu the pair "gn" is essentially a separate letter of the alphabet, with its own sound (like English "ny" or Spanish "ñ") unrelated to those of the other two.  On the other hand, the pair "he" is not a separate letter of the English alphabet, even though it is abnormally common (because of "the", "they", "them", "there") .

For that reason, word statistics are generally more meaningful -- as long one can assume that the spelling and encryption always takes the same plain-text lexeme to the same encoded lexeme.  But even then one must allow for errors and inconsistencies in the data, at all levels -- from the Author himself to the transcriber.

All the best, --stolfi