18-04-2026, 11:24 PM
(18-04-2026, 07:13 PM)quimqu wrote: You are not allowed to view links. Register or Login to view.Even then, the model can still detect line starts and ends quite clearly from character patterns alone. That’s the part I find hard to explain purely as a side effect of line breaking on tokens.
Again, by changing the distribution of token lengths around line breaks, the TLA changes the frequencies of words at those places, and therefore the frequencies of characters and n-grams at those places. See the example of "th" in English.
Thus, even of you delete spaces and join the lines into a single string of characters, the character and n-gram frequencies will still be anomalous around the deleted line breaks.
And this effect is even stronger with the SLA, since the enhanced occurrence of m and reduced occurrence of iin will still be anomalies of the character stream a the places of the deleted breaks.
That is why I still don't consider the LAAFU theory proved. I still don't see evidence that the statistical anomalies around line breaks cannot be explained as side effects of the SLA.
All the best, --stolfi

