16-11-2019, 08:34 PM
(16-11-2019, 06:51 AM)ReneZ wrote: You are not allowed to view links. Register or Login to view.The effect of varying page length, which causes an artificial positive correlation, seems to be effectively eliminated by two different methods:
1) the one proposed by @nablator, by subtracting the expected values based on some average
2) by computing the percentages
While they are different, the resulting correlations are quite similar (certainly equivalent), and any remaining correlation is clearly significant.
The word pairs with ch / Sh largely have a ratio of almost 2:1, but the most frequent pair chedy : Shedy is clearly an exception. This is mostly caused by the biological section, which again turns out to be 'different' in yet one more respect. It has always been seen as the most repetitive of all texts in the MS.
You did not mention that you also removed all zero matches. You warned first that this might be a mistake: "Pages with zero matches (for both words) should probably not be removed, as this is one of the possible results." You warning was correct (see You are not allowed to view links. Register or Login to view.).
Keep also in mind that percentages are bounded by [0, 1], and the underlying assumption of the Pearson Correlation test is that values are normally distributed; these are manifestly incompatible. Moreover, the same percentage values can stand for different page sizes. Therefore it is hard to interpret correlation values between percentages from page length especially since we can not exclude that the page size has any influence.