About the construction of lines in the MS

[quimqu]

I tested a simple question: how predictable is the text, depending on where you look?

Same setup everywhere. Take a fragment, try to predict what comes next, and rank the real continuation against many alternatives. If there is structure, the real one should rank high. If not, it should look random.

First, across lines. I used the end of a line to predict the start of the next one. The signal is weak. The real continuation behaves almost like a random candidate. Top-1 is only a few percent and the average rank sits near the middle. So line-to-line continuity is poor.

Then I did the same inside the line. I split each line roughly in half, always on a space, and used the left part to predict the right part.

Here the behaviour changes a lot. The average rank drops to ~16 out of ~100 candidates, the median is 5, top-1 is ~30%, and more than half of the real continuations fall in the top 5. The real continuation is clearly better than average, although not always the best possible one.

So the contrast is clear. Across lines, almost random. Inside lines, stronger signal.

There are caveats. Frequency effects and position inside the line can inflate predictability, and this does not imply a single deterministic continuation. But the gap is large enough that it is unlikely to be just noise.

One detail matters. The real continuation beats the average candidate, but often not the best one. That suggests multiple plausible continuations rather than a fixed sequence.

There seem to be real local structure inside lines, but it does not carry cleanly across line boundaries, and even locally the system seems to select among compatible variants rather than follow a single path.


Note: I can make all my code available if someone wants to check it.