[Article] update to Zattera's slot machine

[Jorge_Stolfi]

I really have no opinion about the best coverage. Personally, I think the grammar should reproduce all words (maybe with the exception of those with really rare characters like v), otherwise it's not a grammar of Voynichese, but a grammar of some subset of Voynichese, and all conclusions made based on this grammar should carry this footnote with them.

But what is the purpose of the grammar?

Any "ordinary" text in any language will use only a small "random" subset T of that language's lexicon L. For polysyllabic languages, L is typically 100'000 or more.

Thus we cannot know the lexicon L of Voynichese from the VMS text alone.  We can build a grammar -- or, equivalently,  a finite deterministic automaton -- that generates precisely the set T: every word that occurs in the VMS, and no other.  

Such a grammar or automaton can be useful as an efficient way to check whether a word is in T or not (Appel and Jacobson, You are not allowed to view links. Register or Login to view.)  But, because T is a small and random sample of words, it does not give much information about the language, and its complexity c(T) -- the number of states of the automaton, or the number of rules in the grammar -- will be several times  the number #T of words.  That is, in the tens of thousands.

A grammar or automaton for L -- the lexicon of the language -- would be more useful.  But there is no way to get that either. The set L is usually a random subset of W, the set of all words allowed by the language's phonetic and morphological constraints.  Its complexity c(L) will be even higher than c(T).

On the other hand, those constraints are generally fairly simple, so the set W can usually be precisely described by a fairly small grammar or automaton.   The latter may give useful insights about the language. 

However, we cannot categorically recover W from the sample T.  First, a finite sample will generally lack examples of some combinations of phonemes or letters that are valid in W.  Well's novel War of the Worlds contains no word with the letter combination "unstr".  If that novel was the only sample of English we had, should we conclude that this combination is not valid by the English phonetic and morphological rules?  (It is valid, in fact.) 

Second, the set T extracted from a specific text will usually have many "noise" words that are not in W -- spelling errors, onomatopoeias, foreign words, words that have been accidentally split or joined, etc.  The VMS, in particular, must have several hundreds of them.  For example, Well's novel  uses the word "rahnd" instead of "round" to imitate some strong dialectal accent, and the incorrect or old spelling "vallyble".  But neither  "-hnd" nor "-yble" are valid English suffixes.  There is no reliable way to detect such "noise" words; and any grammar that includes all of T will define a set of words larger than the true W.

Therefore, it is pointless to require 100% coverage ("the grammar must generate all the words that occur in the VMS") or 100% precision ("the grammar must not generate any word that does not occur in the VMS").  Or even to try to maximize some combination of the two metrics, like 3*coverage + 5*precision.  A more useful approach must combine those metrics with some measure of the complexity of the grammar.  But the weights of those metrics will be arbitrary...

I think the main idea of loop grammars was to handle all or almost all words while keeping the grammar size manageable.

If the language is polysyllabic, the set W will usually be infinite, and describing it will require a recursive grammar or an automaton with loops.

But everything indicates that the proper words of Voynichese have a very limited size. The longer words are almost certainly cases where two or more words were accidentally stuck together.   In that case W is more naturally described by a non-recursive grammar, or an automaton without loops.

All the best, --stolfi