The structure of the Voynich text and how it may be generated

[Jorge_Stolfi]

I tested both types of artificial line breaking.

Thanks!!

For natural text, I now compute the token-level statistic on the running text itself, without using the artificial lines in the calculation. The artificial lines are used only afterwards, to assign each token to a positional bin such as first, second, middle, penultimate, last. In other words, the line cut no longer changes the token’s measured value. It only changes the group where that value is displayed.

For the Voynich, I do the analogous thing. I reconstruct each real paragraph as one flat token sequence, compute the token-level statistic on that flat paragraph, and only then project each token back to its original line position and line type.

That makes more sense!  

But there is still one problem: we know that the head line and the first few words of each paragraph are special (the "PAAFU anomaly").  That is quite expected, assuming that the paragraphs are what they seem to be (descriptions and uses of the associated plant) and that the puffs are embellished versions of other glyphs that the Scribe was free to use on the head lines.  Therefore, if you use all the lines of re-justified paragraph, you will get anomalies due to the first line.  

To fix that problem, I would discard the first line of the original paragraph, then re-justify the remaining tokens with new page width (in characters, of course). 

By the way, with a variable-width "font" like Voynichese, it should make some difference whether the new page width is specified in mm (when t should count ~3 times as much as e) , in glyphs (counting Ch as 1) or EVA letters (counting Ch as 2).  In English, for example, with TimesRoman font, the line-breaking effect on the frequency of "it" at line end should be at least twice as strong as the effect on "me".

And then we can discuss other details of the Scribe's actual line breaking "algorithm"  that can create additional LAAFU-like effects, even if LAAFU is false...

I have also the plots divided by section, if anyone finds it interesting

Again, mixing all sections will only confuse the issue.  Even if the LAAFU theory is correct, it may work differently in different sections.  I think that the statistics will be easier (or less difficult) to interpret if we consider only one homogeneous section first, like Herbal.  Then later we can look at other sections, each on its own.

For one thing, I expect that PAAFU will be much stronger in Herbal than in Bio.  And it is defintely very strong in the Starred Parags, for the same reason as it is in Herbal.

All the best, --stolfi