15-04-2026, 07:07 PM
After that control, I tried to look inside the signal instead of only comparing AUC.
Not all real breaks are equally well detected, so I split them into “good” (high score) and “weak” (low score), and compared them with interior gaps.
Here the pattern is quite clear.
- Good breaks look very structured. high end, high start, low entropy.
- Interior gaps look very unstructured. low scores, high entropy.
- Weak breaks are in between, but much closer to interior.
So it seems that not all line breaks behave the same. There is a subset that fits very well the “good ending + good beginning” pattern, and another subset that barely fits it.
Then I checked the same idea on the re-justified controls.
They do produce some signal, but it looks much flatter. There are fewer “good” breaks, and the entropy does not drop as much. The structure is weaker and less concentrated.
Also, looking at the actual patterns, in the real text a relatively small set of families explains a large part of the good breaks. In the controls those same families exist, but they are not aligned with the breaks in the same way.
So, it seems that the trivial line breaker explains part of the effect. It creates some bias, especially from word length. But it does not reproduce the strong concentration of “good endings + good beginnings” at the break points.
In the real text, line breaks seem to happen preferentially where both sides look locally well-formed, not just where the line happens to fill.
So it seems that two things are happening at the same time. Lines fill the available space, but they also tend to end and start with specific families of patterns.
That is not what you get if you only re-wrap the same text mechanically.
Not all real breaks are equally well detected, so I split them into “good” (high score) and “weak” (low score), and compared them with interior gaps.
| Section | Group | End | Start | Total | Left H | Right H |
|---|---|---|---|---|---|---|
| Herbal | good | 1.91 | 1.27 | 3.18 | 1.15 | 1.95 |
| Herbal | weak | 0.00 | 0.00 | 0.00 | 3.37 | 3.37 |
| Herbal | interior | 0.27 | 0.31 | 0.57 | 6.24 | 6.19 |
| Biological | good | 2.20 | 2.37 | 4.57 | 2.39 | 2.31 |
| Biological | weak | 0.00 | 0.00 | 0.00 | 3.09 | 3.03 |
| Biological | interior | 0.84 | 0.87 | 1.71 | 5.38 | 5.38 |
Here the pattern is quite clear.
- Good breaks look very structured. high end, high start, low entropy.
- Interior gaps look very unstructured. low scores, high entropy.
- Weak breaks are in between, but much closer to interior.
So it seems that not all line breaks behave the same. There is a subset that fits very well the “good ending + good beginning” pattern, and another subset that barely fits it.
Then I checked the same idea on the re-justified controls.
They do produce some signal, but it looks much flatter. There are fewer “good” breaks, and the entropy does not drop as much. The structure is weaker and less concentrated.
Also, looking at the actual patterns, in the real text a relatively small set of families explains a large part of the good breaks. In the controls those same families exist, but they are not aligned with the breaks in the same way.
So, it seems that the trivial line breaker explains part of the effect. It creates some bias, especially from word length. But it does not reproduce the strong concentration of “good endings + good beginnings” at the break points.
In the real text, line breaks seem to happen preferentially where both sides look locally well-formed, not just where the line happens to fill.
So it seems that two things are happening at the same time. Lines fill the available space, but they also tend to end and start with specific families of patterns.
That is not what you get if you only re-wrap the same text mechanically.