Repetition of words

[Mauro]

I made a test with De Bello Gallico: it has 11030 word types, of which 6314 are hapax legomena. The whole text is 51503 words long, so most of the text (88%) would be culled by your procedure, but only ~43% of the vocabulary.

I understood that Mark considers deleting words that are repeated in consecutive positions.

Exactly. The Voynich manuscript has many words that repeat consecutively, maybe twice or three times in succession. I suspect that such words are very very likely to be null or filler words, so I am curious as to what the Voynich text might look like with all such words removed.

By repeated words I don't mean words that occur more than once in the manuscript; they must be words that are written next to each other in sequence. Once some words are removed there may be other words that are repeated next to each other, so they must be removed and this process repeated until there are no words in the Voynich text that are repeated in succession. I may implement this myself, but I wanted to put it down as a thought first.

Indeed I was unsure. I got it now, sorry. My guess is the text will change very little, if at all, but it's just a guess (I know there are languages where reduplication is an important feature, so it may be different with one of those, but my knowledge does not go beyond what the wikipedia article says).