18-01-2021, 07:12 PM
Hello Marco!
You make some really good points!
I believe we have modified the Wikipedia Corpus slightly since publication, which may explain why your answers are a little different.
Yes, the structure of the text in the Wikipedia Corpus inflates the reduplication rate. The tool I used to compile the texts does not distinguish between title text and content text, as you demonstrated. It also includes a lot of metadata, which I tried to clean as much as possible using a series of regular expressions to capture the most common Wikipedia code snippets, but as Lurker shows there are some I wasn't able to get rid of. These issues are especially relevant for Wikipedia language versions that a) have a small number of articles in total, b) have articles which are short on average, and c) are written in the Latin script (because for other scripts I can just filter out the Latin script metadata). This particularly affects minority languages like Cree and Piedmontese, which also have very basic, formulaic entries.
The Historical Corpus, by contrast, has a much smaller reduplication rate range from 0.0-0.16%, so Voynich is a clear outlier among the historical manuscripts we have. But there may be texts we haven't found that have higher rates of reduplication either because they're in certain genres (e.g. magical encantations) or because the grammar of the language itself uses reduplication more extensively.
All of this is to say that reduplication is an interesting topic that warrants a lot more examination than we were able to give to it in the Review article. Thank you for bringing it up!
Luke Lindemann
You make some really good points!
I believe we have modified the Wikipedia Corpus slightly since publication, which may explain why your answers are a little different.
Yes, the structure of the text in the Wikipedia Corpus inflates the reduplication rate. The tool I used to compile the texts does not distinguish between title text and content text, as you demonstrated. It also includes a lot of metadata, which I tried to clean as much as possible using a series of regular expressions to capture the most common Wikipedia code snippets, but as Lurker shows there are some I wasn't able to get rid of. These issues are especially relevant for Wikipedia language versions that a) have a small number of articles in total, b) have articles which are short on average, and c) are written in the Latin script (because for other scripts I can just filter out the Latin script metadata). This particularly affects minority languages like Cree and Piedmontese, which also have very basic, formulaic entries.
The Historical Corpus, by contrast, has a much smaller reduplication rate range from 0.0-0.16%, so Voynich is a clear outlier among the historical manuscripts we have. But there may be texts we haven't found that have higher rates of reduplication either because they're in certain genres (e.g. magical encantations) or because the grammar of the language itself uses reduplication more extensively.
All of this is to say that reduplication is an interesting topic that warrants a lot more examination than we were able to give to it in the Review article. Thank you for bringing it up!
Luke Lindemann