(26-02-2026, 10:55 PM)quimqu wrote: You are not allowed to view links. Register or Login to view.If the long-range MI gap were mainly an artifact of rare glyphs or over-fragmentation, reducing the alphabet should weaken the effect (this is at least the logical thing: it should behave more like natural languages and reduce the gap, isn't it?).
No idea, really. This is not at all what I have in mind. I should wait to have something positive to report before I comment, more tests to do before that. Migraine today until an hour ago, I tested nothing.
Here is my possibly wrong and very badly explained understanding of the most probable cause of the long-range gap with zero evidence that I will regret posting tomorrow. What is the difference between the distribution of the letter-bigrams (x, y) at (anywhere, anywhere + long distance) and (anywhere, anywhere else)? The long distance of course between the letters x and y. In "normal" texts the language is the same everywhere, so the distribution of letter-monograms is the same everywhere. The two distributions at <anywhere> and <anywhere else> are exactly the same. On the other hand in bigrams (anywhere, anywhere + long distance) the two distributions at position <anywhere> and <anywhere + long distance> remain consistently different over the whole text (or most of it) if there is a (mostly) consistent increasing or decreasing trend in the frequencies of some or all of the letters (many letters and high frequency change to have a big gap, few letters and low frequency change for a small gap). In an evolving text with a direction of evolution, not a completely random walk, distance matters: a (voluntary or involuntary) direction of evolution is the simplest way I can think of that could create the long-range gap, other than an unlikely, I suppose, and epistemologically costly mathematical link between all letters that would need a theory of spooky statistical action at a (long) distance.