16-11-2019, 09:39 PM
(16-11-2019, 08:34 PM)Torsten wrote: You are not allowed to view links. Register or Login to view.You did not mention that you also removed all zero matches. You warned first that this might be a mistake: "Pages with zero matches (for both words) should probably not be removed, as this is one of the possible results." You warning was correct (see You are not allowed to view links. Register or Login to view.).
For the chedy / Shedy pair, there are zero matches in *all* A-language pages, and there are no zero matches, as far as I know, in the B-language pages. The statistics per 'section' of the MS are therefore not affected by this.
(16-11-2019, 08:34 PM)Torsten wrote: You are not allowed to view links. Register or Login to view.Keep also in mind that percentages are bounded by [0, 1], and the underlying assumption of the Pearson Correlation test is that values are normally distributed;
One can compute the correlation coefficient for any set of data without any assumption on their distribution.
You have done exactly the same. Counts are bounded as [0, N] and are nowhere similar to being normally distributed.
(16-11-2019, 08:34 PM)Torsten wrote: You are not allowed to view links. Register or Login to view.... the same percentage values can stand for different page sizes. Therefore it is hard to interpret correlation values between percentages from page length especially since we can not exclude that the page size has any influence.
By comparing the correlation for 'counts' and the correlation for 'percentages', as has been done in this thread, one can see the influence of the page size. This is precisely what has demonstrated that for this particular, and quite interesting, word pair, only the biological section has a significant positive correlation.