[Article] A Hidden Markov Model for the Linguistic Analysis of the Voynich Manuscript

[davidjackson]

I'm afraid I'm becoming very jaded about this sort of research. Great for uni credits no doubt, but not really adding anything to the world of Voynich.

It also illustrates what I was on about in the AI thread about garbage in, garbage out. The author explains:

We started with the same initial conditions as those given in Equations (11) and (12) for the
transition matrix and the distribution of the state t=0. The probability matrix for the observation
states (the Voynich characters) was randomized in the usual way explained in Section 3.1.

In other words, he used initialised the HMM using results found from his previous analysis of an English language translation of Don Quijote (Equations (11) and (12) being biased for the English language text). Meaning the HMM would be biased towards English. Had he used the German version of Shakespeare, no doubt the HMM would have been biased towards German. In third words, he expects to find vowels, and so his model does.

Secondly, the author has made a basic mistake in the transcription he used. He says (P.19):

At this point, it is also be necessary to explain why we chose this particular alphabet [EVA] instead
of the other alternatives. The main reasons are its popularity and the fact that many transcriptions
are available for it. Otherwise, some specialists would argue that some symbol combinations in this
alphabet, such as “ch” and “sh”(corresponding to the so-called “pedestals”) should be considered as
one single character each. On the other hand, the combinations “in” and “iin” are also candidates
for representing letters, although, in some other cases, “i” could be a single character. This is another
problem that computational analyses could help to solve [...]

This way, a transcription of the whole Voynich manuscript has been performed in such a way
that it can be used in computational analysis. [..] In particular, we used Takahashi’s transcription developed in 1999.
Of course, some pre-processing was required before applying the HMM algorithm because this file
includes some information about each line, including the folium number (recto or verso) and the
number of the line within each page of the manuscript. After removing this information, we were left
with a set of EVA characters separated by dots. These dots correspond to the spaces between words in
the original manuscript.

Takahashi's transcription still uses combinations within Eva, noted by capitals. For example, Sh is sh. At no point does the author mention that he changed these combinations into unique characters - what is more, he dismisses such cases and assumes they do not exist in EVA (this is incorrect). Therefore, his HMM will not correctly take these character indications into account, and hence his alphabet is incorrect. If it is case insensitive it will split Sh into s - h (when in the original manuscript this is a different glyph) or if case sensitive it will introduce a non-existent character into the model (S). Either way, it does not correspond to reality and the algorithm is working with an incorrect transcription of the text.