The structure of the Voynich text and how it may be generated

[Jorge_Stolfi]

I have seen one interesting thing: when you take a normal text and cut it into artificial lines, something simple happens. Words at the edges of the line behave differently. The first word has no left context. The last word has no right context. This alone creates a small asymmetry.

You can see it clearly if you measure how predictable a word is from the previous one versus the next one. In natural texts, once you force them into lines, a very regular pattern appears. It’s always the same shape, and it’s not very strong. It comes from the cut, not from the language itself.

As I wrote above, line breaking should produce such anomaly -- but not just because "the previous word has no context".

Again: the trivial line-breaking algorithms: "if the next word fits into the space that remains on the line, write it there, else break the line and write it there."  This algorithm has the effect that the average length of the first word on each line is greater than the global average length, while the average length of the last few words of each line is less than average.  
This effect should cause most word and character statistics be dependent on position along the line.  Including entropy and conditional entropy.

This happens with any text and any language -- as long as the maximum line width is a fixed number of millimeters or of characters (including spaces).   It does not occur if the line length limit is a fixed number of words. 

So the important question is: how did you generate the blue graph? By breaking the running text after N words, or after N characters?

The Voynich behaves differently. It also shows positional effects, but they are stronger and the shape is not the same.  The beginning of the line is much more constrained than expected, and the end behaves differently from what we see in normal texts.

That makes it unlikely that the line structure is just a passive formatting layer. It looks more like something that actively constrains how the text is generated.

The line-breaking algorithm used by the scribe is probably more complicated than the trivial one above.  Those extra details could contribute to the line-start and line-end anomalies of the VMS.  But before we discuss them, please answer the question above.  It is absolutely critical.

All the best, --stolfi