A family of grammars for Voynichese

[Mauro]

Hello Mauro,

If I understand correctly your "chunkification" process, the pattern matching is greedy: this is the first of the three possibilities in the example that I posted earlier on the first page of this thread, and I don't see why it is a better choice that the other two:

If some words are indeed "separable" (as suggested by M. Zattera's article), maximizing the F1 score or any metric is futile: the model is not meant to match "separable" words. Fitting the model to data containing "separable" words is probably what creates the wrap-around effect in the MZ slot sequence: slots 7-11 have 6 glyphs in common with slots 0-2, resulting in multiple possibilities of separation: for example sheolkeedy (in f115v) could be, according to Massimiliano Zattera's 12-slot sequence:
sheol keedy
sheo lkeedy
she olkeedy
These 6 words exist in the VM: it is unclear where the space should be inserted. There could be a rule, for example, to choose the first of the three because it maximizes the length of the first word, but we don't know.

Maybe a non-ambiguous slot sequence, allowing a unique re-parsing from a space-less transliteration, would better capture the partial ordering principle at work in the VM.

You are perfectly right: the pattern-matching algorithm is 'greedy', it will stop at the first match it finds. Actually the algorithm is more sophisticated and I could go on and find all the possible alternative matches too, but I think this would not be much useful. It's one of those things which would be very easy to explain face-to-face just by scribbling on a piece of paper, but it's not easy to explain in written form. I try my best.

It does not matter because, after all, the grammar structure univocally determines the chunk structure (even if there may be many possible alternative 'chunkings'). We could easily re-arrange the grammar to get the alternative chunkings, for instance in LOOP-4 all the 'consonants' [d, k, l, r...] etc. are at the end, so a word such as 'daiin' is chunked as [d] + [aiin]. But I could just put that slot at the beginning, and 'daiin' would become [daii] + [n].

So, what discriminates between one of the two alternatives forms of LOOP-4, that is to say, the different forms of possible chunkings? It's the total number of chunks, Nchunktypes, found by chunkifying each grammar: I actually made a test with the 'consonants' at the beginning (just past the [q, ch, sh, y] slot) and it finds 609 Nchunktypes, so it's slightly worse than the standard LOOP-4 at 606 (but a very viable grammar nonetheless, I guess).  I also tested with 'consonants' both at the beginning and the end and it resulted in ~675 chunks (I don't remember exactly), decidedly worse. That's why in the end I settled for the standard LOOP-4 of post #53.


Addendum. About 'separable' words: I don't think the concept is even meaningful in the context of slot grammars: just duplicate any grammar and it will find all the separable words it had been missing before. Seen from another point of view: it's not that defining a word as 'separable' excuses the original grammar for not having found it