About the construction of lines in the MS

[quimqu]

So, before we can take your results above as evidence of LAAFU, you need to repeat the analysis on text that has been re-justified as discussed before.  Namely, discard the first line of each parag, join the remaining lines in a single token stream, and feed that to the trivial line-breaking algorithm, with maximum line length set to about 62% or 162% of the original average line length -- always counting characters, not words.   

The set of these re-justified parags should be the "null control".  I am quite sure that you will see on this  control text the same kind of anomalies that you see on the VMS.  The question is only whether they will be just as strong, or significantly weaker.

And even if the anomalies in this control text are weaker than on the original parags, that is still not yet evidence of LAAFU.  Because the algorithm used by the scribe is a bit more complicated than the trivial one, and the additional complications add to the line-break anomalies.  So we will have to try to simulate these complications too.

Yes, that was a good point, so I tried exactly that control.

I took the same paragraphs, removed the original line breaks (keeping the first line aside as suggested), joined everything into a single token stream, and re-broke the lines with a simple greedy algorithm based only on character count. I tried three regimes: about 0.62×, 1.0× and 1.62× the original average line length.

Then I did a strict out-of-sample test: train only on real paragraphs, and test on held-out paragraphs, comparing real breaks vs re-justified breaks on the same text.

These are the results.

Section	Dataset	AUC total	End AUC	Start AUC
Biological	Real	0.875	0.743	0.749
Biological	Rejust 0.62	0.555	0.519	0.530
Biological	Rejust 1.00	0.631	0.515	0.555
Biological	Rejust 1.62	0.679	0.527	0.552
Herbal	Real	0.885	0.712	0.756
Herbal	Rejust 0.62	0.731	0.519	0.536
Herbal	Rejust 1.00	0.742	0.526	0.554
Herbal	Rejust 1.62	0.642	0.544	0.563

So yes, the trivial line breaker does create signal. The AUC is clearly above chance, especially in Herbal. So that part of your explanation is correct. But it does not explain everything. The real text is consistently much higher:

Section	Control	Real − control AUC
Biological	0.62×	+0.32
Biological	1.00×	+0.24
Biological	1.62×	+0.20
Herbal	0.62×	+0.15
Herbal	1.00×	+0.14
Herbal	1.62×	+0.24

More importantly, if you look at the actual separation between boundary and interior, the difference is big.

In the controls, boundary vs interior differences are small, around 0.03–0.1.
In the real text they are much larger, around 0.3–0.5.

So the trivial algorithm produces weak preferences, but the Voynich text shows much stronger ones.

Also, this is not only about line length. The biggest gap is in the end and start features. Those are based on local word patterns, not just length. The “fit” term behaves more like you would expect from line-breaking, and I would not overinterpret it.

It seems that the simple character-based line breaker does reproduce part of the effect, but only a fraction of it. When you keep the same text and only change where the lines are cut, the signal drops a lot. The real text still has a much stronger and more consistent boundary structure. So at least in my test, the effect is not just coming from trivial line-breaking.