A family of grammars for Voynichese

[Mauro]

I propose as a ‘figure of merit’ for a slot grammar the total number of unique chunks in the chunkified grammar (Nchunktypes). That is to say: the total number of unique chunks in the second table above. The lower the number, the ‘better’ the grammar. This applies to any grammar, with a loop or without (that would be a grammar where the ‘loop’ repeats only 1 time).

I have some difficulties following your argument, so I hope you won't mind a few dumb questions on my part. How do you define "unique chunks" here? Is it unique per slot or unique in all the slots?

I'm happy to answer any questions, and I apologize for any lack of clarity in what I wrote. More drawings would have been useful, but they take a lot of time to make and then upload, and I kept the explanations as tight as I could to avoid an interminably long post.

With "unique chunks" I mean: count each chunk once, even if it appears in more than one slot of the "chunkified" grammar. That is: count chunk "types". However (see the Note 1 of my post) one could also just count the chunks as they are (so that the same chunk is counted multiple times when it appears in more than one slot): I'm not sure which of the two counting methods is the 'right' one, because both counts (Nchunktypes and, let's say, Nchunktokens) have the same propriety of rejecting both the trivial grammars.

From the way I understand it, the trivial grammar that just lists 25 most common Voynichese characters in 10 slots would have either 25 NChunktypes or 250 NChunktypes with almost 100% coverage. Is this correct?

Yes, right. It would have 25 "Nchunktypes", or equivalently 250 "Nchunktokens". Coverage lower than 100% because it cannot generate words with more than 10 characters.


I add an example here which can maybe make things clearer on how the 'scoring' of grammars work:

Let's say we want to 'score' the grammar of your example. That would be the trivial LL*CS grammar (length * character set, 10 slots * 25 characters in your example). This grammar is rejected because, if we 'chunkify' it, it's very easy to see that each word type becomes a different chunk. So after the 'chunkification' we get the other trivial grammar, the 1*WS (one slot containing every word type in the text), and Nchunktypes is now ~8000 = the number of word tokens in the text (with length <= 10). This is the highest (and worst) possible number of chunks one could get, so the trivial LL*CS grammar goes at the bottom of the 'score' list.

If we want instead to 'score' directly the 1*WS trivial grammar, the 'chunkification' gives us the same grammar (it's invariant under 'chunkification'), and yet again it goes at the bottom of the 'score' list.

So the 'chunkification' works equally well on both the limit cases of trivial grammars (they even get the same score!), which is what we want from a sound 'scoring' method.