I tried keep this post to the bare minimum, cutting a lot of details (you’re free to ask anything, of course). It’s some preliminary results I got exploring the LOOP grammars and what they can do. They look promising, but with all the mandatory caveats needed with the VMS….
----------------
I used the LOOP grammar to divide each Voynich word type in word chunks. I experimented with some variations of the grammar, and in the end I settled on this one:
The loop is repeated up to four times.
Notice this grammar has no pretense of efficiency, and efficiency is not my aim at all. I want to have the simplest possible grammar with a high coverage, which in this case is a hefty 99.28% (on RF1a-n; this excludes word types containing ‘rare’ characters, but I could as well add them so I don’t get bored of typing this…). Only 58 words (on 8031) are missed and they are all with more than 4 chunks, ie. ‘tchodypodar’ would be [t][chod][yp][od][ar]. Just for colour, ‘fachys’ is chunkified as [f][a][ch][ys] and is among the valid word types found by the grammar.
------------------------
So, I divided all the Voynichese word types in chunks following the above grammar, which resulted in 605 different chunks, relatively few (the other variations I tried resulted in more chunk types). Many are quite rare, ie. ‘chai’, used by just one word, ‘ychaies’. I also grouped the chunks in ‘categories’, which is better explained with an example: ‘ched’ uses the slots PE---S, it goes in the same ‘category’ as ‘shed’ and ‘sheed’, which use the same slots. Instead ‘aiin’ uses ---AIS, so it goes in a different category (the ‘Y’ final slot is treated separately and is not considered in the categories, but I may add it in the future, I’ll see). The maximum number of possible categories is 64 (2 raised to 6th power, the number of slots inside the word chunk), but only 51 different ones are actually used by the VMS (some of them extremely rare, ie. the ‘chai’ above is the only chunk in his category), which is mildly encouraging.
This means that the whole VMS (except 58 word types) can be written by using 605 different chunks, to be repeated up to 4 times , which I think is an interesting result. But remember: changing the grammar will result in a different chunk-ification, so I’m surely not claiming that these are the right chunks! They are a possibility, and it doesn’t look too bad to me, for now.
---------------------------
Many criticism can be raised of course, I thought of these:
• 605 chunks is not a big number but it’s also not a small one. It could be that any text could be ‘decomposed’ with analogous results. I find it improbable for natural languages, I would have no clue on how to build a slot grammar for a natural language (their word types are too varied). I also made a sanity check using syllabified natural languages. Results are preliminary, not checked thoroughly and must be taken with a pinch of salt because syllabifying a text is not trivial, but to keep it short, with texts of comparable length (and even shorter ones) I find a least ~1200 different syllables in certain Italian and Latin text (they usually have more), and many more for English. Of course languages written natively with a syllabary will have much less, but they're improbable in medieval Europe.
• Some chunks are almost whole words, so it’s not surprising they can generate Voynich words. Seen from another point of view, I’m approaching the “Trivial 1*WS” grammar (which uses ~8000 symbols) with a grammar with 4 slots (the number of chunks) and 605 symbols (+null). The Ncharset (see post #39) would be rather high but I need more data to calculate this number, it cannot be higher than 4*(605+1) but will be probably lower because I won’t need all the chunks in all the 4 slots.
As another sanity check: the chunks include all the single 'characters'. This would be very bad for the grammar if many words turned out to be chunkified using only single-character chunks but, apart from obvious cases (ie. word types such as ‘d’, ‘dy’, ‘chy’ etc.) this does not seem to be the case.
------------------------
Now what I want to do in the next days is to calculate the frequencies of the chunks on word tokens, ie. P(START)-->’qok’, P(START)-->’ched’… P(qok)-->’eee’ and so on. I think (and hope xD) the number of probabilities different from zero will not be that many (which would be another encouraging thing). Once I’ve done that I should have a 4 slots (unlooping) grammar, one slot for each possible chunk, with 605 symbols (+ null and end-of-word) + frequencies, which should essentialy ‘capture’ all the statistics of the VMS text (except of course long/short-range things like possible correlations between words or the effect of paragraphs etc.), and I can use it as a Markov chain to generate a pseudo-Voynich with random words, then see how it compares to the real VMS on some statistics (and expecially on the vocabulary). I’ll let you know what happens (also in case of a negative, which is always behind the corner with VMS xD).
-----------------
Feel free to post any criticisms, remarks, suggestions, requests for additional informations, whatever.
----------------
The Excel file dump with the word types parsed into chunks, the chunk categories and the list of unique chunks found is here:
You are not allowed to view links. Register or Login to view.
The most used chunk categories are in the following table. Inside each category the chunks lists themselves are +- ordered by frequency (calculated on word types, not tokens), many are quite rare (and would be even rarer over word tokens), but I have no actual frequency stats at the moment (the ordering is an unforseen but welcomed side effect of the method I used to process the VMS lol, it looks right but I’m not 100% sure, some low frequency chunks could be out of order).
# of uses includes all the occurences in the word types, ie. 'olorol' counts as three 'ol'-like chunks.
Certain categories came as a surprise, ie. the 4th-ranked ‘chd/yk’-like and most of all the single ‘o’, which appears much more frequently (but in uncommon word types) than I imagined. An example is ‘oees’ [o][ees], another is ‘dalo’ [d][al][o]. The distribution of the 51 categories of chunks vs. frequency (among word types) is this:
At the moment, I don’t have stats on the chunk types frequency, nor on the distribution of word types vs. number of chunks. More software xD
----------------
I used the LOOP grammar to divide each Voynich word type in word chunks. I experimented with some variations of the grammar, and in the end I settled on this one:
The loop is repeated up to four times.
Notice this grammar has no pretense of efficiency, and efficiency is not my aim at all. I want to have the simplest possible grammar with a high coverage, which in this case is a hefty 99.28% (on RF1a-n; this excludes word types containing ‘rare’ characters, but I could as well add them so I don’t get bored of typing this…). Only 58 words (on 8031) are missed and they are all with more than 4 chunks, ie. ‘tchodypodar’ would be [t][chod][yp][od][ar]. Just for colour, ‘fachys’ is chunkified as [f][a][ch][ys] and is among the valid word types found by the grammar.
------------------------
So, I divided all the Voynichese word types in chunks following the above grammar, which resulted in 605 different chunks, relatively few (the other variations I tried resulted in more chunk types). Many are quite rare, ie. ‘chai’, used by just one word, ‘ychaies’. I also grouped the chunks in ‘categories’, which is better explained with an example: ‘ched’ uses the slots PE---S, it goes in the same ‘category’ as ‘shed’ and ‘sheed’, which use the same slots. Instead ‘aiin’ uses ---AIS, so it goes in a different category (the ‘Y’ final slot is treated separately and is not considered in the categories, but I may add it in the future, I’ll see). The maximum number of possible categories is 64 (2 raised to 6th power, the number of slots inside the word chunk), but only 51 different ones are actually used by the VMS (some of them extremely rare, ie. the ‘chai’ above is the only chunk in his category), which is mildly encouraging.
This means that the whole VMS (except 58 word types) can be written by using 605 different chunks, to be repeated up to 4 times , which I think is an interesting result. But remember: changing the grammar will result in a different chunk-ification, so I’m surely not claiming that these are the right chunks! They are a possibility, and it doesn’t look too bad to me, for now.
---------------------------
Many criticism can be raised of course, I thought of these:
• 605 chunks is not a big number but it’s also not a small one. It could be that any text could be ‘decomposed’ with analogous results. I find it improbable for natural languages, I would have no clue on how to build a slot grammar for a natural language (their word types are too varied). I also made a sanity check using syllabified natural languages. Results are preliminary, not checked thoroughly and must be taken with a pinch of salt because syllabifying a text is not trivial, but to keep it short, with texts of comparable length (and even shorter ones) I find a least ~1200 different syllables in certain Italian and Latin text (they usually have more), and many more for English. Of course languages written natively with a syllabary will have much less, but they're improbable in medieval Europe.
• Some chunks are almost whole words, so it’s not surprising they can generate Voynich words. Seen from another point of view, I’m approaching the “Trivial 1*WS” grammar (which uses ~8000 symbols) with a grammar with 4 slots (the number of chunks) and 605 symbols (+null). The Ncharset (see post #39) would be rather high but I need more data to calculate this number, it cannot be higher than 4*(605+1) but will be probably lower because I won’t need all the chunks in all the 4 slots.
As another sanity check: the chunks include all the single 'characters'. This would be very bad for the grammar if many words turned out to be chunkified using only single-character chunks but, apart from obvious cases (ie. word types such as ‘d’, ‘dy’, ‘chy’ etc.) this does not seem to be the case.
------------------------
Now what I want to do in the next days is to calculate the frequencies of the chunks on word tokens, ie. P(START)-->’qok’, P(START)-->’ched’… P(qok)-->’eee’ and so on. I think (and hope xD) the number of probabilities different from zero will not be that many (which would be another encouraging thing). Once I’ve done that I should have a 4 slots (unlooping) grammar, one slot for each possible chunk, with 605 symbols (+ null and end-of-word) + frequencies, which should essentialy ‘capture’ all the statistics of the VMS text (except of course long/short-range things like possible correlations between words or the effect of paragraphs etc.), and I can use it as a Markov chain to generate a pseudo-Voynich with random words, then see how it compares to the real VMS on some statistics (and expecially on the vocabulary). I’ll let you know what happens (also in case of a negative, which is always behind the corner with VMS xD).
-----------------
Feel free to post any criticisms, remarks, suggestions, requests for additional informations, whatever.
----------------
The Excel file dump with the word types parsed into chunks, the chunk categories and the list of unique chunks found is here:
You are not allowed to view links. Register or Login to view.
The most used chunk categories are in the following table. Inside each category the chunks lists themselves are +- ordered by frequency (calculated on word types, not tokens), many are quite rare (and would be even rarer over word tokens), but I have no actual frequency stats at the moment (the ordering is an unforseen but welcomed side effect of the method I used to process the VMS lol, it looks right but I’m not 100% sure, some low frequency chunks could be out of order).
# of uses includes all the occurences in the word types, ie. 'olorol' counts as three 'ol'-like chunks.
Certain categories came as a surprise, ie. the 4th-ranked ‘chd/yk’-like and most of all the single ‘o’, which appears much more frequently (but in uncommon word types) than I imagined. An example is ‘oees’ [o][ees], another is ‘dalo’ [d][al][o]. The distribution of the 51 categories of chunks vs. frequency (among word types) is this:
At the moment, I don’t have stats on the chunk types frequency, nor on the distribution of word types vs. number of chunks. More software xD