About the construction of lines in the MS

[quimqu]

@nablator, @ReneZ,

I think the concern about spaces is completely valid. If the spacing is unreliable, then anything based on "words" becomes questionable. So, I tried to remove spaces entirely and see what happens.

The test is simply this: I take only paragraph lines, strip all spaces, and learn what line endings look like and what line beginnings look like, but purely at the character level. For this, I used a simple character-level n-gram model trained on line endings and line beginnings, and scored each possible cut by how well its surrounding characters match those distributions. Then I join many lines into one long continuous string and ask: can we still detect where the real line breaks were? The answer is yes, very clearly.

Here are the main numbers:

Dataset	Mean real score	Mean nearby	Mean random	Diff vs nearby	AUC
Same sections (Herbal/Biological)	2.62	-3.07	-3.24	+5.69	~0.93
Other sections	2.95	-2.71	-3.32	+5.65	~0.92

Real line breaks score about 5–6 points higher than nearby non-break positions, and the separation is strong (AUC around 0.9+). Also, in about 99% of cases, the real cut looks more like a line break than the surrounding positions.

So even after removing all spaces, the signal is still there, and it is strong.

This matters for the point you both raise. Maybe spaces are not fully reliable. Maybe what we call "words" are not real words. Maybe something like daram should indeed be split as dar am in some cases. But whatever the correct segmentation is, the line structure does not disappear when spaces are removed.

What the model is actually picking up are not “words” but character patterns. Certain endings keep appearing at line ends, like …dy, …dain, …dal, …dar, …ol. And certain patterns prefer the start of a line, like sol-, sor-, qot-, qok-, dch-, dsh-. In real examples you see things like:

…daroly | salche
…hdyoly | dcheyl
…eyqoly | dsheyq

These are not clean lexical units. They are just recurring fragments that behave differently at boundaries.

Even if spacing is imperfect, and even if tokens are not true words, there is still a real constraint at the level of characters that marks where lines end and begin. That constraint is strong enough to recover line breaks from a continuous no-space string.

-----------

As an extra check, I ran exactly the same character-level model again, but this time keeping spaces as ordinary characters instead of removing them. The result is also clear: the signal is already strong without spaces, so the line-break effect does not depend on tokenization. But if spaces are kept as characters, detection becomes even stronger. I am not saying that all spaces are correct, or that the spaced units are necessarily true words. It only means that the transmitted spacing is not random noise. It carries additional structural information that helps identify real line breaks.

Version	Dataset	Mean real score	Mean nearby score	Diff real-nearby	AUC vs nearby
No spaces	Held-out Herbal + Biological	2.62	-3.07	+5.69	~0.93
Spaces kept as characters	Held-out Herbal + Biological	3.46	-3.31	+6.77	~0.94-0.95
No spaces	Other sections	2.95	-2.71	+5.65	~0.92
Spaces kept as characters	Other sections	3.93	-3.01	+6.94	~0.94-0.95

Removing spaces does not kill the effect, and keeping them improves the model, which suggests that the spacing, even if imperfect, still reflects some real structure in the text.