11-12-2019, 12:39 AM
(10-12-2019, 03:32 PM)MarcoP wrote: You are not allowed to view links. Register or Login to view.the fact that Currier B adds new word types to Currier A is certainly central to the whole phenomenon. I don't think that it can explain everything, since there are fluctuations in the frequencies of words that appear everywhere that must have another explanation.
For instance the occurrences of chol/cheey vary from 192/15=12.8 in Herbal_A to 12/36=0.3 in Bio. The frequency of chol in Herbal_A is 2.5%: comparable with the frequency of the most frequent word in English. Even if tokens belonging to the new B-types are frequent, I don't think they are enough to explain how a word drops from 2.5% to 0.2%.
Indeed, a token dominating a paragraph, page or section might be rare or missing on the next one.
An interesting example for this pattern is the usage of <You are not allowed to view links. Register or Login to view.> (see You are not allowed to view links. Register or Login to view.). Jorg Stolfi has described the usage of <qokeey> within the stars section in terms of three distributions:
"If a paragraph contains the word <qokeey>, there is a 38% chance that the next paragraph will contain the word <qokeey>.
If a paragraph contains the word <qokeey>, there is a 40% chance that <qokeey> occurs more than once.
If the current word is <qokeey>, there is a 6% chance that the next word will be <qokeey>" (You are not allowed to view links. Register or Login to view.).
The general rule behind this pattern is that "high-frequency tokens also tend to have high numbers of similar words." (Timm & Schinner 2019, p. 6). With other words <You are not allowed to view links. Register or Login to view.> occurs preferably in close vicinity of tokens with high structural similarity (see Timm & Schinner 2019, p. 3):
okey okeey qokeey qokeedy word count
------ ------ ------- -------- -----------
Herbal (A) 4 7 12 8,087
Pharma (A) 11 24 21 2,529
Astro 4 6 4 2,136
Cosmo 1 6 6 4 2,691
Herbal (B) 6 8 9 9 3,233
Stars (B) 16 96 159 137 10,673
Biological (B) 12 19 89 153 6,911