25-04-2026, 03:24 PM
Jorge,
I tried to replicate your previous experiment on the Herbal and Biological sections, with a similar setup (glyph classes, no spaces, filtering short lines).
The results are qualitatively the same: within lines there is short-range predictability (around 20–25% at d=1, decaying quickly), while across line breaks it drops to ~2–3%, essentially baseline. This suggests a very strong break effect, consistent across both sections.
Separately, I also looked at token-level effects:
– Line starts and line ends are both clearly special, with fairly symmetric signals.
– However, the dependency between the end of one line and the start of the next is weak.
– Around <->, the effect weaker than at the end and beginning of lines but it is asymmetric: the token before the split is more “end-like” than the token after it is “start-like”.
So, to summarize:
– there is local structure within lines
– line breaks seem to disrupt that structure quite strongly
– positional effects (start/end, and before <->) are real but not identical
As you noted, part of the within-line predictability is driven by specific patterns like q→o, and their uneven distribution (e.g. near line starts) can explain some of the effects. Also, strong token families at line ends likely contribute to the boundary signals.
At this point I am not assuming any particular explanation. Maybe it is something old and that has been discussed hundreds of times, but for me, discovering that there seems to be a break from line to line is new, and is something that I will take into account if someday I try to test any theroy of my own.
Anyway, thank you for your testing and thoughts!
I tried to replicate your previous experiment on the Herbal and Biological sections, with a similar setup (glyph classes, no spaces, filtering short lines).
The results are qualitatively the same: within lines there is short-range predictability (around 20–25% at d=1, decaying quickly), while across line breaks it drops to ~2–3%, essentially baseline. This suggests a very strong break effect, consistent across both sections.
Separately, I also looked at token-level effects:
– Line starts and line ends are both clearly special, with fairly symmetric signals.
– However, the dependency between the end of one line and the start of the next is weak.
– Around <->, the effect weaker than at the end and beginning of lines but it is asymmetric: the token before the split is more “end-like” than the token after it is “start-like”.
So, to summarize:
– there is local structure within lines
– line breaks seem to disrupt that structure quite strongly
– positional effects (start/end, and before <->) are real but not identical
As you noted, part of the within-line predictability is driven by specific patterns like q→o, and their uneven distribution (e.g. near line starts) can explain some of the effects. Also, strong token families at line ends likely contribute to the boundary signals.
At this point I am not assuming any particular explanation. Maybe it is something old and that has been discussed hundreds of times, but for me, discovering that there seems to be a break from line to line is new, and is something that I will take into account if someday I try to test any theroy of my own.
Anyway, thank you for your testing and thoughts!