15-04-2026, 12:33 PM
(This post was last modified: 15-04-2026, 12:34 PM by Jorge_Stolfi.)
(15-04-2026, 08:25 AM)quimqu wrote: You are not allowed to view links. Register or Login to view. The null control shows that this does not survive random relabelling. So the effect looks real. I think the safest formulation is this: in Herbal and Biological, line breaks are statistically structured rather than arbitrary.
Good, but, again, that is an expected (and even already verified) consequence of the trivial line-breaking algorithm (provided that the margins are defined by mm or character count, not by word count). It does not insert line breaks at random, but in a way that strongly depends on the lengths of the words before and after the break. As a result, the first word after a line break tends to be longer than average, and the last 1-3 words before a line break tend to be shorter than average. And if the word length distributions are different, the same will almost certainly be true of any other other statistic, character- or word-based.
So, before we can take your results above as evidence of LAAFU, you need to repeat the analysis on text that has been re-justified as discussed before. Namely, discard the first line of each parag, join the remaining lines in a single token stream, and feed that to the trivial line-breaking algorithm, with maximum line length set to about 62% or 162% of the original average line length -- always counting characters, not words.
The set of these re-justified parags should be the "null control". I am quite sure that you will see on this control text the same kind of anomalies that you see on the VMS. The question is only whether they will be just as strong, or significantly weaker.
And even if the anomalies in this control text are weaker than on the original parags, that is still not yet evidence of LAAFU. Because the algorithm used by the scribe is a bit more complicated than the trivial one, and the additional complications add to the line-break anomalies. So we will have to try to simulate these complications too.
All the best, --stolfi

