Measuring Long-Range Structure in the Voynich Manuscript

[quimqu]

To myself (and this is full my theroy), we could say that the area between the full text and the shuffled text is the effect that the word ordening has to the MI (so how the natural language links words and characters in a long term).

I am trying to understand the asymptotic behavior: your MI(d) curves seem to converge to a different value in word-shuffled texts than in "raw" (original) texts, right? If so, it seems very counterintuitive: there should be no predictability at all at large distances. For the "raw" texts because obviously words are preferentially followed by other words, but this fact tells you nothing about the words on the next page or 10 pages later. For the word-shuffled texts it is often characters of the same word that are picked together at low distances so there can be some predictability, it diminishes with distance and goes down to zero as soon as the distance exceeds the length of the longest word.

The explanation must be in the way MI(d) is calculated or displayed. Pardon the very basic question: how is it defined? Is it this formula?

The mutual information of two jointly discrete random variables X and Y is calculated as a double sum:

You are not allowed to view links. Register or Login to view.

That's right, Nablator. It is this formula. It is essentially the mutual information between characters separated by exactly d positions, averaged over all such pairs in the corpus.

Mutual information at distance d does not measure whether we can predict words on the next page. It simply measures whether knowing one character slightly changes the probability distribution of another character located d positions away.

It is true that in a perfectly random and fully independent sequence, MI(d) would go to zero as d increases. But real texts are not random or stationary. They contain repeated words, recurring morphemes, topic clusters, stylistic patterns... All of these create weak but measurable statistical dependencies, even if they do not correspond to meaningful prediction across pages.

In the case of word shuffled texts, word structure is preserved, but the order of words is destroyed. Ths is why most medium-range structure collapses. However, because the same words are still present, and because some words are repeated or clustered, a small residual dependence can remain. I have observed that in natural languages, the  residual dependence is "the same" as for the raw texts (at high d) but, that for generated text like Torsten's (without a meaning) this residual dependence is different. Why does this sappens? Well, I guess (I repeat, I just guess), that the remaining MI(d) which is the total words of the corpus keeps a sort of natural meaning  that does not depend on the order of the words. In torsten's text, this meaning disappears.

I know it is hard to explain or understand... I am just guessing and making some suggestions about the findings as I haven't seen any literature that talks about these differences.

Please note, this is not semantic prediction at long distances, but simply a reflection of the overall statistical structure of the corpus.