About the generation of similar words

[quimqu]

I'll leave it here for today, but first I wanted to look at one last thing: if there was any pattern in how the burst types alternate within the pages.

The intuition was clear. Perhaps the text was alternating types in a regular manner, or following increasing or decreasing sequences. But the results point in a different direction.

Code:

Pattern class | Mean share same | 0.514 up_jump | 0.181 down_jump | 0.181 up_1 | 0.087 down_1 | 0.084

More than half the time, the type stays the same. And when it changes, it often does so in leaps and bounds, not smooth progressions.

If we look at slightly longer sequences, it becomes even clearer:

Code:

Pattern class | Mean share flat_3 | 0.315 nondecreasing | 0.201 nonincreasing | 0.200 aba | 0.201

There is no strong alternation of the type A B A B, nor are there linear paths. What does appear a lot is the pattern A B A: the system leaves one type, but quickly returns to it.

In other words, the dominant behavior is not alternation, but persistence with small local excursions. The text seems to move around a “state” and make short detours before returning to it.

And this fits very well with everything that has been coming out: strong internal structure within the bursts, real sequential dependency, but changes that do not follow simple rules.

And now, for today I'll leave it here. More tomorrow  (I will reply if you have comments but for today the coding is finished)