About the construction of lines in the MS

[quimqu]

So it seems that not all line breaks behave the same. There is a subset that fits very well the “good ending + good beginning” pattern, and another subset that barely fits it.

It could be that, unsurprisingly, some words fit in the available space without shortening. Is the average length of the "good" words smaller than the average length of the "barely fits" words?

That’s a good point, and I already checked it before.

If this was mainly a fitting problem, you’d expect the “good” endings to be clearly shorter than the ones that barely fit. But that’s not what we see. The average length is basically the same. Finals are around ~4.29 characters, and the matched interior controls are also ~4.29. Even if we focus only on strongly end-like tokens, it’s something like ~3.8 vs ~3.9, so still very close.

At the same time, the difference in end-likeness is huge. Finals are around +0.95, while controls are around −0.78. So length is almost identical, but the behavior is completely different.

That makes it hard to explain this just as a packing or fitting effect. Short words alone don’t explain why certain tokens are so strongly preferred at line ends. It looks more like something acting at the level of specific variants or families, not just length. Remember that the model learns the families with a training set and test them with lines not seen. This for herbal and biological sections. But when we test the model to the rest of the sections (not seen) the results are also pretty good.

---

Have you used the words before/after "<->" in your before/after linefeed data or not? If not, what happens if you add them?

I will try to answer this question this evening or tomorrow.

---

Perhaps it suggests that the meaningful content is in the central part of each line, with filler words added at the beginning and end.

No idea. I think this finding can help checking for new hypothesis.