16-02-2026, 03:10 AM
(16-02-2026, 12:29 AM)Dunsel wrote: You are not allowed to view links. Register or Login to view.If there’s a natural-language example where a very high-frequency digraph shows this kind of behavior, I’d genuinely be interested to see it. It would help clarify whether what we’re seeing here is unusual or not.
Your "control" examples seem to be all inherently homogeneous, in that every page it likely to use the same common words, and every bigram occurs in many of those words. Thus it is not surprising that the plots have that shape -- which is probably predictable under the assumption that each text is produced by a single first-order Markov model. (The "Chymical" treatise could be heterogeneous, but from the plots it does not seem to be.)
You are not allowed to view links. Register or Login to view. is an English Herbal (Culpeper's) and You are not allowed to view links. Register or Login to view. is the graph of the number of occurrences of "rk" on each "page", defined as 100 consecutive lines. Each bar is a "page". As you can see, there are some "pages" with 9 occurrences of "rk", mostly from the word "bark" in entries about trees; and several sets of several consecutive pages with no "rk" at all, which seem to be about herbs, not trees. Maybe it is not as dramatic as the case of ed on the VMS, but should suggest the explanation for it.
(The number of occurences above is actually the number of lines with 'rk'. Multiple 'rk' on the same line are counted as one.)
(The source file has many @-lines which specify titles and such. I forgot to remove them, but they should not make much difference.)
What is the average size of a page in your plots? Culpeper's herbal has on average ~3900 letters per "page".
All the best, --stolfi