14-11-2019, 02:23 PM
Thanks to @farmerjohn for the additional comparisons. This shows that the same effect is also observed for other pairs than the ch* / sh* pairs.
To Marco: working with percentages would remove the contribution to the correlation that just comes from page length. In the imaginary case that I used to illustrate the effect (arbitrarily picking words), the expected correlation would be 0.
Pages with zero matches (for both words) should probably not be removed, as this is one of the possible results.
Finally, this is still not on the topic of David's question....
To Marco: working with percentages would remove the contribution to the correlation that just comes from page length. In the imaginary case that I used to illustrate the effect (arbitrarily picking words), the expected correlation would be 0.
Pages with zero matches (for both words) should probably not be removed, as this is one of the possible results.
Finally, this is still not on the topic of David's question....