About the construction of lines in the MS

[quimqu]

@Jorge, @nablator, I ran the tests with the <-> spacing (images intra lines).

The internal split points marked with <-> do show part of the same effect seen at true line endings, but more weakly. I trained the model only on normal paragraph lines from Herbal and Biological, and then tested the tokens immediately before and after <->. If those split points behave like true visual interruptions, the token before <-> should look more like a line ending, and the token after <-> should look more like a line beginning.

That is exactly what happens on the left side of the split. In Biological, the token before <-> has a mean end-like score of +0.42, compared with -0.89 for ordinary interior tokens, although still below true line endings at +0.84. In Herbal, the pattern is very similar: +0.45 before <->, versus -0.60 for interior tokens, again below true line endings at +0.82.

The right side of the split shows a weaker version of the same pattern. In Biological, the token after <-> has a mean start-like score of +0.07, compared with -1.01 for interior tokens, but still far below true line starts at +1.02. In Herbal, it is -0.10 after <->, versus -0.63 for interiors, again much weaker than true line starts at +1.13.

So the best summary is that <-> does not behave exactly like a real line break, but it is not neutral either. The token before the split is clearly more line-end-like than a normal interior token, and the token after the split is slightly more line-start-like than a normal interior token. The effect is therefore real, but partial, and stronger on the left side of the split than on the right side.