How multi-character substitution might explain the voynich’s strange entropy

[quimqu]

It is tempting to assign "entropy" to a single text sample (which I might call a "micro-message").  But the sequence of characters is known, so all of the probabilities are unity;  insofar as it can be said to have an entropy, the value is exactly trivially zero.  The meaningful entropy that we discuss here should be attributed to the text generator ("macro-messenger"), as a logarithmic measure of how many different micro-messages it is capable of producing.  An approximate value can be calculated from one micro-message, but the accuracy of approximation depends on its length.  I expect that there are formulas in the literature that relate n-mer order, sample size, and systematic error.  The only way to evaluate high-order conditional entropies of the VMS author is to discover more text that it produced.

Thanks for your interesting reply — and yes, you're absolutely right about the challenges in interpreting high-order n-gram entropies with limited text. I just wanted to clarify that in my analysis I was actually working with perplexity, rather than entropy directly. That’s the metric I’m more used to, especially in the context of language modeling, as it's more intuitive: it roughly reflects how “surprised” a model is when trying to predict the next character. As you know, perplexity is just the exponential of the entropy (assuming log base 2), so the trends are equivalent — just represented differently.
I completely agree that the drop in entropy (or perplexity) at high n values is partly a numerical artifact, especially because long repeated sequences are rare. This naturally leads to lower perplexity, simply because the model starts memorizing the few long words it has seen before — essentially overfitting.
That said, what caught my attention — and what motivated this line of inquiry — was something peculiar I observed when comparing Voynich text to natural language: in a perplexity-vs-n plot, the Voynich text showed a distinct “bump” or bulge at mid-range n, whereas real language tends to show a much smoother, almost linear decline. Look at the graphic:

This anomaly made me wonder: could certain types of ciphers produce similar bumps when applied to real language? That became the basis of the experiments I’m currently running. Surprisingly, most ciphers (like simple substitutions or transpositions) still result in a smooth drop in perplexity. Only one type — what I initially called a homophonic cipher, though it’s more accurately a multi-character substitution cipher — generated a similar bump to the one I see in the Voynich text.
So, I’m not claiming this is conclusive by any means, but the similarity in the perplexity curve between the Voynich and this kind of ciphered text definitely raised some interesting questions.