About the construction of lines in the MS

[quimqu]

The TLA algorithm may create anomalies also on the second token of the line, if there is correlation between the lengths of successive tokens.

Jorge, I think your explanation via a line-breaking mechanism (TLA + spacing adjustments) makes sense for part of the effect, especially the general bias toward shorter tokens near the end of the line. My gradient result is indeed consistent with that kind of mechanism.

However, there are two points where I think the data go beyond a pure TLA explanation.

1- the signal is not only about length. The gradient is computed on a family-based “end-likeness” score trained out-of-sample, and what increases toward the end is not just shortness, but membership in specific token families. A pure TLA would primarily act on length distributions, but here we see structured preferences at the level of variants.

2- I tested the compression hypothesis directly. If line-end effects were mainly driven by local compression or abbreviation, we should see that final tokens correspond more often to longer interior forms. But we do not. In fact, finals show slightly fewer such relations than matched interior controls. So the effect is not behaving like a systematic shortening process.

Your point about spacing, ambiguous boundaries (e.g. after r or s), and local expansion or contraction is very plausible, and could explain part of the anomalies. But the results suggest that line-end behavior is not reducible to spacing and line breaking alone. There seem to be a positional preference for certain families or variants that are built towards the end of the line.