The 'Chinese' Theory: For and Against

[kckluge]

the distribution of the number of words between instances of words with a length-normalized edit distance below some threshold [...] is an incredibly significant statistical signature of the Voynich text that any theory about how the text is generated needs to reproduce.

Can you be more specific?

Sure. See Schinner, Andreas (2007) 'The Voynich Manuscript: Evidence of the Hoax Hypothesis', Cryptologia,  31:2, 95 - 107 (specifically, p 100-103). Key passage: "The Levenshtein distance of two character strings is an integer ranging from 0 (exact match) to the maximum of the two string lengths (no similarity), denoting the number of elementary edit operations necessary to make both strings equal. Mapping this number to the interval [0,100]  yields a 'percentage of dissimilarity' for two tokens. In Figure 3, the similar token repetition distance distribution Pn for the VMS compared with normal texts is presented. Here n denotes the number of other tokens between two similar ones, i.e., n = 0 corresponds to the situation of two alike tokens in immediate vicinity. Two words are considered 'similar' if their dissimilarity as defined above is less or equal to 30%; it turns out that the precise value (+/-10%) of this threshold changes Pn only quantitatively, not qualitatively."

It's for Stolfi to address that issue -- quantitatively -- in the context of explaining/defending his theory.  [...] The "explanation" would be to show that the same anomalies that you see in the SPS are spelling system, and incidence of errors.

Can you see those anomalies in You are not allowed to view links. Register or Login to view.?

Not sure where the "The 'explanation'...of errors" bit came from -- probably a Ninja editing quote nesting error.

Having incorporated code that replicates Schinner's analysis into my Swiss Army text stats analyzer (although not in the version I released on the Ninja), I can check at some point. Given that I'm currently trying to help someone get a start-up off the ground it may not be very soon. Just based on eye-balling it, I'll be surprised if it has a distribution that's similar to the Voynich text. 

...I'm acutely aware that this is the "Chinese" theory thread (and not, for instance, the (AFAIK non-existant) "verbose cipher" theory thread), bur moderators and readers please bear with me because what follows will, in fact, wrap back around to being on topic...

I don't mind.  Besides, any argument you present for why the "Chinese" Origin theory (COT) is wrong will be in-topic.

Appreciated, I just want to avoid the thread getting dragged too far off topic.

3) results in a critique of Stolfi's approach here

First, note that the type-to-token ratio (TTR) of a text in any language usually depends on the text size.  You are not allowed to view links. Register or Login to view. (which may be just a consequence of Zipf's law) says that the size L of a a text's lexicon (number of "word types") is related to size N of the text (number of tokens) like L ≈ K sqrt(N), where K may depend on the language, style, topic, etc.  

So, when comparing TTRs of different texts, it is absolutely important to trim them to the same number of tokens.  Or instead compute the apparent constant K = L/sqrt(N)  instead of the ratio L/N.

Understood, and the word count of the Roger Bacon excerpt was picked to match the sample of Voynich text I was comparing it to. Given that You are not allowed to view links. Register or Login to view. appears to show that the Voynich text follows Heap's Law, I'll need to add testing for fit to it into my Swiss Army text analysis program...

Second, spelling and transcription errors can inflate L while having little effect on N.   Mandarin uses only 1300 syllables (~18%) out You are not allowed to view links. Register or Login to view. by the known phonetic constraints of the language. That means that changing one phoneme in a random syllable will, with high probability, increase the lexicon size L -- even if the change respects the phonetic constraints. 

Thus, when comparing lexicon sizes, it may be prudent to exclude word types that occur only once or twice.  Or maybe even more, depending on the rate of errors.

I have no problem believing there are scribal and transcription errors in the Voynich text -- "There is supposed to be a space before every EVA 'q', even though there isn't" is a hill I am more than willing to die on. I think if you're positing rates that are unusually large compared to known rates of scribal errors in manuscript texts then that's potentially a problem (and way back in this thread I suggested using rates of scribal errors in Greek mss. copied by an "illiterate" scribe as a reasonable proxy for the Voynich).

I am also fully on board with needing to try to clean the data -- in the experiments I've done with algorithms from the literature for induction of regular grammars with only positive training examples I always exclude hapax, and generally exclude types that only occur twice, even though that massively reduces the number of training samples.

As for "spelling and transcription errors can inflate L while having little effect on N", look, I'm taking a very firm methodological position here that I'm very comfortable defending. Don't tell me "can", show me "does." Look though the right end of the telescope. If you think the Voynich Mss. text is Chinese, don't start by translating Voynichese into Chinese. Start by showing how to translate Chinese into Voynichese. And yes, I understand that you don't actually think it's Chinese per se, and per my comment below I understand that you can't follow this path given the nature of your claim. From my point of view, that's a problem.

Third, one must not forget that statistics -- word and character frequencies, correlations, Zipf's and Heap's law, etc -- are a property of a text, or a specific collection of texts -- not of a language.  There is no such thing as "the frequency of 'e' in English", "the most common word in Latin", "the Heap K of Mandarin".  Even if the dialect and spelling are fixed, those statistics are greatly affected by topic, style, nature of the text, cost of vellum, etc.  One can have a meaningful and grammatically correct text in English with 50'000 tokens and only 100 word types -- that does not use the word "the" even once.

It's a strawman to suggest anyone doesn't understand that these are things that have some variance around mean values. I think you exagerate the extent to which "those statistics are greatly affected by topic, style, nature of the text, cost of vellum, etc." (and the fact that hill climbing n-gram stats decrypters work as well as they do would appear to be an existence proof that they are not). If you are familar with studies quantitatively analyzing how letter and word frequency stats differ between (say) a corpus of chemistry and physics texts and a corpus of mystery and spy novels, I'd genuinely love to see them.

All of which leads to wrapping this back around to discussing Stolfi's theory [...]. It's not *impossible* that Stolfi has stumbled into a solution with his approach, but [...] I think it is far more likely that he has fallen prey to the siren call of the "crib" that has lured so many mariners sailing on the treacherous seas of the Voynich Mss to their doom.

Well, what can I say?  I think that the matches I have found are quantitatively categorical.

I know you do, and believe me, nothing would make me happier than if you or anyone else came up with something I saw as a compelling solution, if only because then I could stop wasting time on the damned thing. It's not like I'm making any progress.

Note that my solution is very different from most other proposals. I don't know what is the language and the encoding, but I claim that I have identified a specific plaintext (the SBJ) whose structure  matches the SPS much better than could be expected by chance.  Specifically, I have a small dictionary (the "cribs") such that those words occur in the SBJ with spacings that are very closely proportional to the spacings of the corresponding words in the SPS.  

And there's the rub, isn't it? If you don't have any idea what the language and encoding are, then you can't follow the methological path I'm advocating. To be clear, that doesn't mean you're wrong, it just makes it harder for you to convince me you're right. 

As for the (statistical) significance of the spacings of the cribs, there is a limit to how closely I've been following the thread after the initial "daiin" match, and I don't want to get into why I'm unconvinced of the significance of the "daiin" spacing. 

What he *should* be doing -- again, IMHO -- is taking texts in one or more SE Asian languages (dealer's choice), assigning some scheme for representing them with Voynich glyphs, simulating whatever "confused ignorant scribe" error processes he thinks are there, and then showing that you wind up with something that -- quantitatively, and for all the "greatest hits" properties (including "babble-like sequences of similar words") -- looks like Voynichese.

Well, you have that text above to play with.  Do you see "babble-like sequences" in it?

At the end of the day, the question isn't "am I willing to put the time into doing it?". It's "are there other higher priority things I need to be using that time for?". Unfortunately, right now the answer is yes, there are. I've pointed you at the Schinner paper, it has figures showing the difference between the behavior of the Voynich text and the samples of natural language text he looked at. If I find myself having a half hour or so window of free time, I'll try to run the analysis.

All the best, --stolfi

Ditto. If it turns out you're right, no one will be happier for you than I will. -- Karl