[split] Arguments for the text being generated

[Jorge_Stolfi]

This is GPT verbatim but it summarises my research

Thanks for posting this summary!  But let me summarize my objections too:

Corollary: folios do have local vocabulary structure, but it is not well described as selecting words from a fixed page-specific list.

Not sure what this means, but we expect that a herbal (which is ~1/3 of all parags text in the VMS) would have a more or less fixed core vocabulary V0 that occurs uniformly over all pages (apart from sampling error), plus maybe some vocabularies T1, T2,.. (like "tea" or "poultice")  that depends on the type of plant and occurs only in  certain subset of the pages, plus some vocabulary  P1, P2,... (like the plant name) that occurs on only in one page.  Except that the vocabularies V0 and Tk may be different for Herbal A and B. 

Whereas a materia medica (which is at last a strong possibility for the Starred Parags section) should not have well-defined Tk, and the vocabularies Pk (the names of the remedies) should be specific for each parag.

Isn't this the case?

Voynich words have a strong internal grammar. They are not arbitrary strings of glyphs. Some glyph sequences are effectively forbidden.

No kidding...

Some apparently dramatic glyph rules turn out to depend on transcription conventions.

Who would have guesses this...  

The extra predictive value of third-order context changes sign or drops below our resolution threshold ... Voynich token formation needs at least second-order context.

Word structure models seem to imply this result.  But they also imply that much longer contexts are important.  For instance, to check or enforce the rule "at most one gallows per word" one needs one bit of information that depends on all preceding glyphs, possibly 6 or more.

The shape and class of the previous token matter reliably. Its exact identity may sometimes add a small extra effect. Adjacent words are linked ... Corollary: neighbouring Voynich words are not sampled independently. the next word depends partly on the form of the previous word. Most next-token predictability comes from structural features rather than remembering the exact previous word. lag 1 is much stronger than lags 2 to 4.

Doing the analysis this way may be misleading.  It may well be that it is in fact the exact identity of the token that matters, but the class and shape alone give good predictions because all statistics will be dominated by the most frequently occurring tokens. 

For example, in English narrative or discursive prose the most common words by far are "the", "and", and "of".  If one builds a next-token predictor based on the first two letters of the current word, one should get quite good results; and knowing the rest of the word should not add much. But that is because the first two letters already identify the whole word with significant accuracy. 

Could this be happening in the tests discussed above?

The end of one word helps predict the beginning of the next.

You mean, like a word ending with "we" in English is  followed by the word "are" more often than expected by chance?

"q" is not simply an independent prefix. It participates in the internal grammar of the word.

The first part is true.  The second part could be stated more informatively as "if q (or qo) is assumed to be a prefix, it can be attached only to certain words, not to any word at random"

The length of an "i" run helps determine how the run ends.

Yes.  But more, 

• the statistics of "ir" are more similar to those of "iin" than to that of "in",
  
• the statistics of "iir" are more similar to those of "iiin" than to those of "iin"
  
• the statistics of "iiir" are more similar to those of "iiiin" than to those of "iiin"
  

the latter is just saying that "iiir" and "iiiin" are both essentially absent.
And this pattern can be explained by the "r" in these endings being a misreading of "in" by the Scribe, considering that the shapes can be quite similar in hurried handwriting.

Voynich words form dense edit-distance families. One-edit relationships are especially important. changing a single glyph is one of the main ways the system creates related forms. These families are productive. Corollary: the system is not just memorising a list of words. It appears to reuse form-building rules.

Again, the observation is correct but the corollary is an unwarranted conclusion.  Those results can be explained by variations in the spelling of words, which can be due to several causes including inherent spelling flexibility (like one could have written "been" as "bin" or "bene" in 1600's English, in the same text), scribal errors, fading ink (that could easily turn Sh into Ch, Ch into ee, y into a or o, etc.)

pairs roughly of the form "Xy ↔ Xdy" behave differently in matched contexts.  Corollary: adding or removing "d" is associated with a systematic change in how the form behaves.

How strange.  Like adding or removing final "-s" in English?  Or adding and removing final "-g" in Mandarin?

Corollary: the vocabulary is better understood as a network of transformations than as a flat list of unrelated words... Different operators move together.... But they do not collapse neatly into one hidden master state...  Corollary: the system seems to have several interacting grammatical dimensions.

Can't these observations be explained as the result of spelling variations, as proposed above?

Currier [A/B languages] is a real distributional distinction, but we cannot yet say with confidence whether it represents dialect, author, cipher setting, or something else.

That is: we have not made any progress on this question in the last 50 years.

Section also changes the grammar... Herbal, Biological, Pharmaceutical, Stars and other sections do not simply use different vocabularies. They alter the probabilities within a shared system. ... the manuscript seems to use a shared grammar that is locally adjusted by section.

So there is a non-trivial grammar common to the whole VMS, but sections with vastly different topics and text structure use some distinctive phrases and sentence structures?  Who would have guessed... 

(Seriously, isn't this a strong argument against "gibberish"?)

Position inside a physical line matters  - Line beginnings, middles and endings behave differently. Line openings and line endings favour different forms.  Corollary the physical line has its own boundary rules.

The differences are real, but the corollary is misleading as stated.  The encryption camp seems to have  taken it as meaning that the encryption method is somehow reset or modified around line breaks (the LAAFU theory).  But recently it was pointed out that the "anomalous" statistics around line breaks can have several other explanations, including 

• the fact that line breaks in any running text have a clear tendency to occur before longer words;
  
• the high likelihood that m is an abbreviation for iin (and maybe other endings) that the scribe could use when space was tight;
  
• the tendency for word spaces to be omitted in the final part of a line;
  

AFAIK it has not been shown that these effects cannot suffice to explain the observed anomalies around line breaks. 

there is no evidence for one hidden state controlling an entire line.

Which kinda goes against the LAAFU theory, no?

Recent reuse [of a word] on the same page improves prediction. local repertoire is a genuine feature of the manuscript. reuse probability changes with distance.

Again, what a surprise...

Paragraph position is a major factor  The first line of a paragraph is special.  paragraph beginnings represent a distinct grammatical state. the manuscript has structure above the level of the individual line.

And this too? 

There is a separate vertical grammar of line openings ... the vertical grammar is richer than a single three-state cycle.

This claim should be more detailed.  What are the transition probabilities? Do they depends on distance from parag start? How do they change with section?  Do they extend across parag  breaks in the Stars section? How accurate is the prediction?

A model that reproduces only Voynich-looking words, edit-distance families, Currier differences, line effects, or opener cycles is not enough.The simplest architecture consistent with the evidence so far is:

• a global form grammar
  
• plus locally conditioned choice among related forms
  
• plus section and Currier effects
  
• plus within-line word-to-word and recurrence rules
  
• plus physical line-position constraints
  
• plus paragraph state
  
• plus a mostly separate vertical line-opener process.
  

As you know, I have a simpler "architecture" that seems to explain all these effects.  But that goes in another thread...

All the best, --stolfi