(30-03-2018, 08:57 PM)ReneZ wrote: You are not allowed to view links. Register or Login to view.Hi Marco, this is very interesting.
I haven't yet have time to fully digest it, but I have an initial question:
how did you compute the 'expected' occurences of X as part of 'X Y X Z'?
Intuitively, for the more frequent cases of X, the actual number should clearly exceed the statistically expected number, but this intuition could be wrong.
Hello Rene,
I am glad you find this interesting and I hope I haven't made any major error.
I computed the expected number in the same way as I did for expected repetition / reduplication:
N*N/TOT
Where N is the number of occurrences of word "X" and TOT is the total number of words in the text.
If a word occurs 50 times in a 1000 long text, it occurs 5% of the times.
5% of the 50 occurrences are expected to match the pattern:
50*50/1000=50*5%=2.5
This is likely overestimated, because I include in this number some combinations I actually excluded from the "actual" counts (e.g. X X X Z): but I think the impact of this imprecision is small.
As a VMS example: 'daiin' occurs 845 times in the ms, which has a total of 37718 words.
I computed the expected number of occurrences of the pattern daiin Y daiin Z as 845*845/37718=18.9
As you can see from the graphs, it seems that different languages behave differently. The most frequent word exceeds the expectation in Latin, but not in Italian and English.

