An Artificial Construction

[oshfdk]

I don't mean to be a pain, but that doesn't solve the problem of word boundaries.... I think this applies to Chinese characters and, of course, VMS, because the spaces here are most likely not normal spaces—at least not all of them... So what exactly are u comparing here? That's the interesting question...

This is a good question. The approach I took should work regardless of the script conventions used. Since it can match Chinese characters (no spaces, non phonetic) and pinyin (phonetic, with spaces), it should work regardless of what the spaces and abbreviations are, as long as they are used according to some universal scheme and not randomly. If the spaces are predictable they will be merged with other tokens.

Here's side by side the computation run on an English text and on Voyniches with spaces and with all spaces removed before processing. As you can see, for the most frequent 5 token combinations there is no much difference. English shows a lot of unbalanced prefix-suffix pairs (mostly just missing from the text), Voynichese only shows a few.