About the construction of lines in the MS

[quimqu]

Sorry, I was confused by your use of the word "learn".   What do you mean by that? 

Hello Jorge,

by "learn" I just meant estimating empirical frequencies from the training set. 

In this version, I am not using a neural net or a full character n-gram model. Instead, I map tokens to coarse "families" (based on length bin and edge substrings), and estimate how often each family appears in three positions: line start, line end, and line interior.

From that I compute two scores:
end score = log2(P(family | line end) / P(family | interior))
start score = log2(P(family | line start) / P(family | interior))

These scores are then used in several tests to evaluate whether candidate boundaries look more like real line breaks than controls.

So the basic idea is: does the token on the left look unusually line-end-like, and does the token on the right look unusually line-start-like, relative to interior tokens?

Given the prvious m→iin experiment, it looks like the signal is not driven mainly by that specific abbreviation. The structure stays strong even when we remove it, and SLA still doesn’t reproduce it. So it seems that whatever is creating the boundary signal involves multiple patterns, not just m (or n). I can try to check whether specific characters or short patterns are disproportionately contributing to the effect. I'll let you know...