About the construction of lines in the MS

[quimqu]

Are you sure this result is real, and not just an artifact of sampling error?

Jorge,

I see your point about the sample size and exact pairs. That’s why I didn’t rely on counting repeated pairs in my tests.

What I did instead was compare the real continuation with many alternatives, both inside lines and across line breaks.

The result,  is quite consistent:

• inside the line, the real continuation tends to score clearly better than alternatives
  
• across lines, the real continuation behaves almost like a random choice
  

So the difference shows up even without needing repeated pairs.

I also checked this in a few ways to make sure it’s not an artifact: using large sections (Herbal and Biological), repeating with different random splits, comparing against controls with similar frequencies... And one important point: I removed spaces and line breaks completely and worked only at the character level. Even then, the model can still detect where line breaks were. So the signal is not just coming from token boundaries or spacing. Even in that setup, the same pattern can be seen: strong structure locally, but very weak continuity across lines.

I agree that small samples can be misleading with pair counts, but the effect I’m seeing does not depend on that. It appears in setups where sparsity is not the limiting factor.

P.S. I don't know if @nablator have checked my code. There might be errors. In the last version all the tests run from the beginning to the end. It takes a bit less than an hour to run all the tests in Kaggle, but I am absolutely open to double check anything.