22-06-2018, 12:14 PM
(22-06-2018, 10:30 AM)bi3mw wrote: You are not allowed to view links. Register or Login to view.(22-06-2018, 08:27 AM)MarcoP wrote: You are not allowed to view links. Register or Login to view....
I guess that an additional test you could run is randomly scrambling word order in one of the two pages (without altering words) and see how much the results change.
...
Thanks Marco,
Your suggestion to test shuffled text against original text is very interesting. I repeated the experiment twice and the result was the same every time: the number of exact and similar hits is halved.
Thank you, bi3mw, very interesting again!
If this figure is confirmed in a number of observations, your analysis can proceed in several directions.
One could think of how to discriminate between the 50% matches that appear to be random and those that must be explained. A simple idea could be ignoring the matches that involve frequent words (those are more likely to be coincidental).
Given the uncertainty in position (due to uncertainty in word spaces) one could also allow some tolerance in position (e.g. considering matches with up to a 5% difference in position). In my opinion, this could be more informative than allowing word-variation via Levenshtein: I think that focussing on perfect matches could be useful as a first step.
In general, if the phenomenon turns out not be illusory, it could be an effect of local concentration of words inside pages: e..g some words tend to occur in the top half of a page and others in the bottom half. Ideas like this can of course be independently tested by measuring the average page-position of each word through the whole manuscript (or through consistent subsections of the manuscript).
I will certainly follow the developments with attention!

